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Preface 


The  purpose  of  this  thesis  was  to  determine  which 
documents  cited  by  DOD-STD-2167A  are  required  for  the 
maintenance  of  a  software  system.  This  information  is 
needed  to  ensure  that  maintenance  personnel  can  readily 
access  the  ideas  and  requirements  used  by  the  original 
designers . 

The  research  made  use  of  existing  information  as  well 
as  a  survey  of  those  personnel  involved  in  the  acquisition 
and  maintenance  of  software  systems.  The  result  is  a  clear 
picture  of  where  these  documents  fit  in  during  the 
maintenance  phase  of  the  software  life  cycle.  If  properly 
utilized,  this  information  provides  a  first  step  in 
reducing  the  tremendous  cost  of  software  maintenance. 

In  preparing  this  thesis  I  have  been  assisted  by  many 
people.  I  am  greatly  indebted  to  my  advisor,  Capt  David 
Luginbuhl ,  and  reader,  Lt  Col  Patricia  Lawlis,  for  their 
assistance  in  finding  information  and  communicating  my 
thoughts.  I  also  wish  to  thank  Dr.  Guy  Shane  for  hi  j 
assistance  in  developing  the  survey.  Finally,  I  wish  to 
thank  my  children,  Scott  and  Amy,  for  understanding  that 
supper  may  be  late. 


Timothy  S.  McArthur 
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Abstract 


This  study  investigated  the  requirement  for 
documentation  during  the  maintenance  phase  of  the  software 
life  cycle.  Without  proper  documentation,  maintenance 
personnel  are  not  able  to  effectively  understand  the 
design  of  a  software  system  and  spend  many  hours 
performing  design  recovery  before  any  type  of  maintenance 
can  be  performed. 

A  review  of  existing  information  showed  a  wide  range 
of  opinion  among  the  experts  in  the  field  of  software 
maintenance  and  was  not  conclusive.  A  survey  of  Air  Force 
software  maintenance  personnel  was  then  conducted  to 
determine  the  need,  availability,  and  tailoring  results  of 
the  documents  listed  in  DOD-STD-2167A.  The  analysis  of 
this  survey  data  showed  that  11  of  the  18  documents  are 
required  for  maintenance.  These  include  the 
specifications,  design  documents,  user’s  manuals  and 
programmer's  manuals.  Plans,  reports  and  background 
information  should  be  provided  to  the  management  of  the 
maintenance  organization  for  initial  planning  of  the 
maintenance  activity.  The  only  documents  deemed  not 
needed  during  the  maintenance  phase  are  those  that  are 
eventually  incorporated  into  other  documents  that  will  be 


vx 


provided  to  the  maintenance  organization.  The  response  to 
tailoring  questions  was  insufficient  and  could  not  be  used 
for  a  conclusive  analysis. 

Providing  only  the  documents  required  by  software 
maintenance  personnel  will  reduce  the  time  required  to 
understand  the  software  and  result  in  lower  costs  during 
software  maintenance. 


Vll 


DOCUMENTATION  REQUIREMENTS  FOR 


SOFTWARE  MAINTENANCE 


I.  Introduction 


General  Issue 

In  1987  the  cost  of  software  maintenance  within  the 
U.  S.  Government  was  reported  to  be  approximately  $3.5 
billion  (Caron,  1987).  By  1990  this  figure  had  exceeded 
$17  billion  for  the  DOD  alone  (Ferens,  1990).  As  the 
importance  of  software  in  weapon  systems  continues  to  grow 
and  budgets  continue  to  decrease,  the  DOD  must  find  ways 
to  reduce  the  amount  of  money  spent  on  software 
maintenance . 

The  primary  impediment  to  software  maintenance  is 
software  complexity  (Harrison  et  al ,  1982).  In  recent 
years  researchers  have  spent  a  great  deal  of  time 
attempting  to  quantify  software  complexity.  While  their 
metrics  may  not  yet  offer  a  clear  solution  to  the  high 
cost  of  maintenance,  they  have  provided  an  insight  into 
the  software  characteristics  that  are  impacted  by 
complexity  (Harrison  et  al ,  1982):  understandabi 1 ity , 
modifiability,  and  testability.  If  these  characteristics 
can  be  improved,  the  complexity  of  the  system  will  be 
reduced  and  the  cost  of  maintenance  will  go  down. 
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Software  maintainers  currently  spend  47%  to  62%  of 
their  time  just  trying  to  understand  the  program  (Haley, 
1989).  Since  the  organization  that  maintains  the  software 
is  generally  not  the  same  one  that  developed  it,  proper 
documentation  is  essential  to  communicating  the  abstract 
ideas  and  module  interactions  that  exist  in  a  program.  If 
the  thoughts  of  the  original  developers  could  be 
effectively  communicated,  the  maintainers'  tasks  of 
modifying  and  testing  the  software  would  be  much  easier 
( Schneidewind ,  1987:304). 

Specific  Problem  Statement 

The  largest  cost  driver  in  software  maintenance 
(program  understandability)  can  be  significantly  reduced 
by  supplyi.",^  proper  documentation  to  the  software 
maintainers  (Martin  ^nd  McClure,  1983).  Ironically,  the 
current  problems  with  documentation  exist  at  both 
extremes:  too  much  documentation,  and  not  enough 

documentation.  When  provided  all  the  documentation 
recommended  by  DOD-STD-2167A,  the  maintainers  must  spend 
hours  sifting  through  enormous  piles  of  paper  to  find  the 
information  they  need.  Even  for  simple  projects,  there 
can  be  a  considerable  amount  of  documentation  (Buckle, 
1984).  On  the  other  hand,  the  developing  organization 
often  fails  to  produce  specific  documentation  needed  for 
maintenance  because  that  documentation  is  not  needed 
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during  the  development.  In  both  cases,  it  is  not  clear 
what  documents  are  useful  or  should  be  produced  (Rubey, 
1985)  . 

Research  Objectives 

The  purpose  of  this  research  is  to  provide  software 
maintenance  organizations  within  the  Air  Force  with  a 
basis  for  determining  documentation  requirements  for  the 
maintenance  of  software  systems.  The  information  obtained 
provides  a  justification  for  documentation  requirements 
during  software  acquisition  and  a  guide  to  providing  only 
the  right  documentation  to  maintenance  personnel . 

The  research  will  investigate  the  following  questions: 

a.  What  documents  (from  DOD-STD-2167A)  are  needed  to 
maintain  software? 

b.  Is  the  needed  documentation  being  provided? 

c.  Does  the  documentation  contain  too  much 
information  that  is  not  needed? 

d.  Is  other  documentation  (not  listed  in 
DOD-STD-2167A)  needed? 

Scope  of  the  Research 

This  research  will  address  only  Mission  Critical 
Computer  Resources  (MCCR).  This  does  not  include  software 
for  Automatic  Data  Processing  Equipment  (ADPE)  which  is 
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used  for  processing  personnel  records,  finance 
information,  and  other  such  business-oriented 
applications.  MCCR  includes  software  that  is  essential 
for  the  deployment  of  a  weapon  system.  This  includes 
software  necessary  for  the  operation  of  aircraft  systems, 
support  equipment,  training  equipment,  and  simulators. 

The  study  will  focus  on  software  that  is  maintained 
organically  by  the  Air  Logistic  Centers  (ALCs)  and  the 
software  that  is  currently  being  purchased  by  the  System 
Program  Offices  (SPOs). 

An  investigation  of  the  current  state  of  software 
maintenance  documentation  will  be  presented  and  existing 
information  will  be  investigated  to  provide  an  initial 
assessment  of  what  documentation  is  required.  Finally,  an 
evaluation  of  the  opinions  of  those  people  within  the  Air 
Force  who  actually  maintain  software  will  be  provided. 

This  will  provide  the  information  needed  to  categorize  the 
documentation  and  ensure  only  the  specific  documents 
needed  for  maintenance  are  delivered. 
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II.  Background 


Definitions 

Before  starting  a  discussion  of  the  problems  of 
software  maintenance,  it  is  important  to  clearly 
understand  what  software  maintenance  is.  The  IEEE  (1983) 
defines  software  maintenance  as: 

"The  modification  of  a  software  product  after 
delivery  to  correct  faults,  to  improve  performance 
or  other  attributes,  or  to  adapt  the  product  to  a 
changed  environment." 

This  divides  software  maintenance  into  three  distinct 
areas  which  are  commonly  called  corrective  (to  correct 
faults),  perfective  (to  improve  performance)  and  adaptive 
(to  adapt  to  a  changed  environment)  maintenance.  This 
thesis  does  not  differentiate  among  the  three  types  of 
maintenance . 

Another  aspect  of  software  maintenance  is  the 
maintainability  of  the  software.  Maintainability  is 
defined  (IEEE, 1983)  as  the  ease  with  which  maintenance  on 
a  software  package  can  be  performed  and  is  usually 
measured  in  terms  of  software  complexity.  As  stated 
earlier,  though,  current  measures  of  complexity,  and 
therefore  maintainability,  are  immature  and  do  not  provide 
much  insight  into  the  true  nature  of  the  software. 
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Lastly,  we  must  consider  a  definition  for 
documentation.  IEEE  (1983)  defines  documentation  as: 

"Any  written  or  pictorial  information  describing, 
defining,  specifying,  reporting  or  certifying 
activities,  requirements,  procedures,  or  results." 

This  definition  is  very  general,  and  this  generality  will 

soon  be  shown  to  be  a  hindrance  to  the  overall  software 

maintenance  process. 

Why  Documentation  for  Software  Maintenance  is  Important 
When  maintenance  is  performed  on  a  software  system, 
the  maintainer  must  first  understand  all  of  the  workings 
and  interactions  of  the  program.  Because  of  the 
additional  effort  needed  to  understand  the  software,  a 
change  made  during  maintenance  may  cost  as  much  as  ten 
times  what  it  would  have  cost  during  design 
(U.  S.  Congress,  1989:9).  Unfortunately,  not  all  faults 
or  future  requirements  can  be  found  or  anticipated  during 
design,  so  the  maintenance  phase  will  play  a  vital  role  in 
the  life  cycle  of  the  software.  Preparations  for  that 
maintenance  must  begin  early  in  the  life  cycle  of  the 
software  system. 

Traditionally,  preparations  for  maintenance  do  not 
begin  until  the  system  is  delivered,  because  maintenance 
is  wrongly  viewed  as  a  phase  completely  separate  from 
software  development  (Martin  and  McClure,  1983).  This 
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results  in  delivery  of  software  that  is  often 
unmaintainable  and  generally  undocumented.  The  lack  of 
planning  for  maintenance  is  not  a  vicious  attempt  by  the 
developers  to  undermine  the  maintenance  effort,  but  more  a 
lack  of  understanding  by  the  development  organization  of 
the  impacts  of  their  efforts.  Software  is  no  longer  (and 
maybe  never  was)  the  easily  changed  subset  of  a  large 
system;  now  it  is  often  the  major  cost  driver  and  basis  of 
the  entire  system.  In  the  rush  to  field  a  system,  the 
development  organization  narrows  its  focus  to  the 
development  at  hand  and  often  doesn't  consider  future 
requirements  to  change  the  software.  The  requirements  for 
software  maintenance  are  ignored  and  the  documentation 
that  is  needed  is  not  provided  because  the  need  to  change 
software  has  been  underestimated. 

Proper  documentation  is  the  physical  representation  of 
the  ideas  and  processes  that  were  used  to  implement  a 
software  system  (Evans  et  al . ,  1983)  and  it  should  enhance 
the  readability  and  usability  of  the  programs  (Martin  and 
McClure,  1983).  Because  software  is  intangible  and 
abstract,  documentation  is  essential  to  communicate  the 
purpose  of  the  code  (Fox,  1982).  Adequate  documentation 
is  useful  during  both  development  and  maintenance  of  the 
software.  New  personnel,  in  either  phase,  will  be  better 
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able  to  understand  the  program,  thus  easing  a  portion  (and 
therefore  cost)  of  both  the  development  and  maintenance 
phases . 

Many  argue  that  good  documentation  is  irrelevant 
because  it  will  not  be  maintained  with  the  software  and 
will  quickly  be  out  of  date.  This  is  actually  an 
indication  of  a  weakness  in  the  defined  software  process 
rather  than  an  indication  of  the  usefulness  of  the 
documentation.  Glass  and  Noiseux  make  this  point  very 
clear  (1981:35): 

"The  intelligent  software  maintainer  should 
realize,  almost  intuitively,  that  keeping 
documentation  up  to  date  is  a  vital  part  of  any 
software  change  activity.  It  is  a  task  analogous 
to  the  craftsman  in  another  field  keeping  his 
toolbox  organized  and  his  tools  oiled." 

Unfortunately,  it  is  often  management  that  pushes  the 
maintainer  away  from  good  documentation  practices,  because 
the  same  pressures  that  existed  during  the  development 
phase  still  exist  during  the  maintenance  phase. 

Problems  With  Software  Maintenance  Documentation 

Although  the  software  aspect  of  most  new  acquisitions 
is  the  major  cost  driver,  procurement  practices  still 
focus  on  the  hardware  (U.  S.  Congress,  1989:4-5).  Even 
during  the  maintenance  phase,  the  hardware  is  still  seen 
as  the  embodiment  of  the  system,  with  the  software  as  a 
minor  subset  that  is  wrongly  perceived  as  something  that 
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is  easy  to  add  or  change.  This  focus  on  the  hardware, 
according  to  Debra  Haley  (1988:22)  of  the  Air  Force 
Coordinating  Office  for  Logistics  Research  (AFCOLR) ,  " .  . 

.  threatens  to  undermine  US  security  by  consuming  more  and 
more  defense  appropriations." 

Because  we  continually  underestimate  the  complexity 
and  importance  of  the  software  aspect  of  a  weapon  system, 
both  in  development  and  maintenance,  we  usually  get  to  a 
point  where  it  becomes  clear  that  schedules  cannot  be  met. 
To  keep  the  project  moving,  a  common  response  is  to 
proceed  with  the  technical  tasks  and  ignore  the 
documentation  (Evans  et  al . ,  1983:60).  Even  when  the 
project  is  progressing  smoothly  the  documentation  is 
generally  given  a  low  priority  on  the  schedule  of 
deliverables  (Arthur  and  Stevens,  1989:42).  One  common 
practice  is  to  complete  the  software  development,  release 
it  to  the  field,  and  then  produce  the  documentation  that 
is  needed  (Arthur  and  Stevens,  1989:40).  The  obvious 
problem  with  this  is  that  by  the  time  the  development  is 
complete,  the  developers  can't  remember  all  the  details  of 
what  was  done.  Additionally,  by  the  time  the 
documentation  arrives,  the  organization  that  maintains  the 
software  (which  is  generally  not  the  same  as  the 
developing  organization)  has  already  made  changes  to  the 
system.  This  can  result  in  very  little  consistency 
between  the  documents  and  the  code  (Antonini  et  al . , 
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1987:91).  Since  the  documentation  is  obsolete  before  it 
is  delivered,  the  only  reliable  source  of  information  that 
exists  is  the  program  code  (Antonini  et  al . ,  1987:91). 

This  problem  is  not  unique  to  the  Department  of  Defense, 
though.  Even  in  industry,  many  programs  exist  with  no 
documentation  (Martin  and  McClure,  1983). 

As  stated  in  chapter  1,  the  opposite  problem,  too  much 
documentation,  also  exists.  This  is  often  due  to  the  very 
general  nature  of  the  definition  of  documentation.  Since 
the  definition  basically  covers  anything  that  can  be 
written  or  drawn  about  the  software,  it  provides  no  limits 
to  what  is  useful.  As  some  organizations  realize  the 
importance  of  software  support  requirements,  they  attempt 
to  improve  the  quality  of  the  product  by  adding 
unnecessary  requirements  for  additional  documentation 
(Haley,  1989:22).  What  results  is  voluminous  amounts  of 
detailed  documentation  that  add  to  the  maintenance  cost  by 
requiring  a  major  documentation  update  effort  each  time 
the  software  is  modified  (Martin  and  McClure,  1983). 

The  military  standard  that  addresses  documenting  a 
software  project,  DOD-STD-2167A,  lists  18  documents  needed 
for  the  life  cycle  of  the  software  system.  These  18 
documents  correspond  to  a  maximum  effort  in  software 
development  (Acquisition  Logistics  Division,  1989). 
Although  not  all  documents  are  needed  for  every  phase  of 
the  software  life  cycle,  each  is  needed  in  some  phase. 
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Many  of  the  documents  that  are  necessary  for  the  original 
development  are  not  needed  during  the  maintenance  phase, 
and  their  delivery  to  the  maintenance  organization  may 
only  detract  from  the  understanding  of  a  program.  When 
the  maintainer  has  every  bit  of  information  produced 
during  software  development,  the  needed  information 
becomes  lost  in  piles  of  paperwork.  This  practice  has 
resulted  in  maintainers  spending  47%  to  62%  of  their  time 
just  trying  to  understand  the  documentation  (Haley, 
1989:22).  Like  any  other  craftsman,  the  software 
maintainer  will  eventually  sift  through  all  the  documents, 
find  the  information  that  is  useful,  and  maintain  only 
that  documentation.  If  the  proper  documentation  had  been 
delivered  to  begin  with,  this  major  effort  would  have  been 
avoided.  One  of  the  costliest  problems  in  software 
maintenance  is  determining  what  documentation  is  needed 
(Arthur  and  Stevens,  1989). 

Proposed  Alternatives  to  Proper  Documentation 

Attempts  to  develop  cost  effective  tools  and  methods 
for  the  maintenance  programmer  have  met  with  very  little 
success  (Landis  et  al . ,  1988:66).  Many  of  these  tools  and 
methods  rely  on  reverse  engineering  the  design  or 
documentation  from  source  code  input  (Antonini  et  al . , 
1937:91-100,  Fay  and  Holmes,  1985:194-202,  Landis  et  al . , 
1988:66-73).  Reverse  engineering  is  a  common  practice  in 
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the  world  of  hardware,  but  in  attempting  to  transfer  that 
technology  to  software,  two  major  differences  must  be 
considered.  First,  the  goals  are  different.  When  reverse 
engineering  a  piece  of  hardware,  the  goal  is  generally  to 
duplicate  the  system.  With  software,  duplication  is  a 
simple  task  usually  left  to  a  clerk  rather  than  an 
engineer.  Reverse  engineering  of  software  is  performed  to 
understand  the  original  design  so  that  it  can  be  modified 
(Chikofsky  and  Cross,  1990:14).  The  second  difference 
deals  with  the  level  of  abstraction  of  the  system.  A 
piece  of  hardware  is  a  concrete  representation  of  an 
understood  design,  but  software  is  often  a  representation 
of  some  abstract  thought  process  (Fox,  1982:204).  Because 
of  these  differences,  no  way  has  yet  been  found  to  extract 
total  knowledge  of  a  design  from  the  program  code 
(Antonini  et  al . ,  1987:91). 

Conclusion 

Good  documentation  is  essential  for  quality 
maintenance  of  software.  It  is  needed  during  the 
development  phase  to  counteract  personnel  turnover,  and  it 
is  certainly  needed  during  the  maintenance  phase  to 
communicate  the  abstract  ideas  of  the  designers.  The 
software  maintainers  may  also  need  documentation  that 
wasn't  needed  during  software  development. 
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There  are  many  reasons  adequate  documentation  is  not 
acquired  in  a  timely  manner: 

a.  The  cost  of  documentation  often  accounts  for 
as  much  as  15%  of  the  total  project  cost 
(Buckle,  1984). 

b.  As  schedules  slip,  programmers  are  often 
relieved  of  documentation  tasks  to  speed  up 
programming  tasks  (Landis  et  al . ,  1988:66). 

c.  The  need  to  change  a  program  is  not 
understood  (Martin  and  McClure,  1983). 

Because  of  the  high  cost  of  software  maintenance,  it  is 
clear  that  the  development  and  maintenance  phases  must  be 
treated  as  being  at  least  equal  in  importance  (Caron, 
1987:36).  The  software  process  model  must  consider  the 
maintainability  of  the  system  in  all  phases  of  the  software 
life  cycle  and  ensure  the  tools  needed  to  do  the  job  are 
available.  The  most  feasible  way  to  do  this  is  to  ensure 
that  the  proper  documentation  is  delivered  with  all 
software . 

This  leaves  us  with  a  very  important  unanswered 
question:  what  documents  need  to  be  provided?  As  will  be 

shown  in  the  next  chapter,  even  the  experts  do  not  agree  on 
the  answer.  This  thesis  attempts  to  answer  the  question  in 
a  general  nature  based  on  the  requirements  of 
DOD-STD-2167A. 
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Ill . 


Existing  Information 


Conversion  of  Expert  Opinion  to  DOD-STD-2167A 

DOD-STD-2167A  establishes  standard  requirements  for 
software  development  that  apply  throughout  a  software 
system's  life  cycle.  The  standard  provides  the  means  for 
establishing,  evaluating,  and  maintaining  software  and 
software  documentation.  Included  in  DOD-STD-2167A  is  a 
list  of  eighteen  Data  Item  Descriptions  (DIDs)  which  spell 
out  the  format  and  content  of  the  documents  required  for 
the  life  cycle  of  a  software  system. 

This  section  reviews  articles  by  authorities  in  the 
field  of  software  maintenance  to  evaluate  their 
perspectives  on  documentation  requirements  for  software 
maintenance.  These  experts'  opinions  are  summarized  and 
mapped  to  the  DIDs  listed  in  DOD-STD-2167A  for  comparison; 
then  a  tentative  list  of  prioritized  documents  is  produced. 
The  order  of  presentation  in  this  section  does  not  imply  a 
ranking  of  expert  opinion. 

Each  article  or  book  is  listed  by  author,  with  a  list 
of  that  author's  view  of  what  documents  are  required  for 
maintenance.  The  equivalent  DOD-STD-2167A  document  has  been 
added  in  parenthesis,  based  on  the  content  specified  in  the 
DID  for  each  document.  Some  of  the  DOD-STD-2167A  documents 
may  cover  more  than  one  of  a  given  author's  requirements 
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and  a  given  document  may  contain  much  more  than  is  required 
by  the  author.  Tailoring  of  these  documents  is  not  covered 
in  this  section. 

A  list  of  acronyms  is  provided  in  appendix  F. 

Acquisition  Logistics  Division  (ALDl.  The  Supportable 
Software  Acquisition  Guide  (ALD,1989)  was  the  product  of 
the  Supportable  Software  Acquisition  Working  Group 
chartered  by  ALD.  The  working  group  consisted  of  project 
officers,  engineers,  and  software  maintainers  with 
considerable  experience  in  the  problems  of  development  and 
maintenance  of  software.  The  guide  states  that 
".  .  .  delivered  documentation  must  include  at  least: 

1.  Description  of  what  the  software  will  do  (SRS, 

IRS) 

2.  Description  of  how  the  software  will  do  it  (SDD, 
SPS) 

3.  Performance  measures  the  software  must  meet  to 
show  it  meets  requirements  (SRS) 

4.  Description  of  any  support  software,  software 
engineering  environment,  or  integration  support 
facility  used  in  developing  and  testing  software 
(SSDD,  CRISD) 

5.  Detailing  on  using  items  in  4  (SSDD,  CRISD) 

6.  Specific  inputs,  scenarios,  and  acceptance 
criteria  for  each  performance  measure  (STP) 

7.  Details  on  results  of  each  test  (to  check  if 
tests  rerun)  ( 3TR) 

8.  Details  on  what  was  coded,  tested,  and  delivered 
(VDD) 

9.  Details  on  interfaces  and  methods  used  by 
software  packages  to  communicate  with  each  other 
(  IDD) 

10.  Total  system  requirements  documents  (SRS,  IRS) 

11.  Description,  data,  test  procedures,  test  results, 
etc.,  from  system  integration  tests  (STD,  STP, 

STR) 

12.  Description  of  how  an  operator  uses  the  software 
and  interprets  its  results  (SUM,  CSOM)" 
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Rubey .  The  Guide  to  the  Management  of  Software  in 
Weapon  Systems  (Rubey,  1985)  was  developed  under  contract 
to  the  Army  and  the  Air  Force  as  an  introduction  to 
software  management  within  the  DOD .  The  guide  lists  ten 
documents  required  for  software  maintenance. 

1.  Requirements  Specifications  (SRS) 

2.  Design  Specifications  ( SSDD .  IDD,  SDD) 

3.  Test  Plan  (STP) 

4.  Test  Procedures  (STP) 

5.  Test  Report  (STR) 

6.  As-Built  Description  (VDD) 

7.  System  Specification  (SPS) 

8.  Software  Interface  Specif ication  (IRS) 

9.  Software  Data  Base  Specification  (SDD) 

10.  Users  Manual  (SUM,  CSOM) 

Rubey  states  that  the  first  six  documents  listed  are 
also  required  for  development,  while  the  last  four  listed 
are  additional  requirements  for  maintenance. 

Buckle .  In  Managing  Software  Projects  (1984:99-102), 
J.K.  Buckle  says  that  the  maintenance  programmer  is  not 
interested  in  what  the  original  intent  was,  but  rather  in 
what  was  actually  implemented.  He  lists  five  documents 
that  are  necessary  for  software  maintenance: 

1.  Design  Description  (SSDD,  IDD,  SDD) 

2.  Hierarchical  Design  Description  (SDD) 

3.  Design  and  Implementation  Documents  for  Support 
Tools  (SSDD,  CRISD) 

4.  Compiler  Listings  (SPM) 

5.  Testing  Documents  (STP,  STD,  STR) 

Fox .  Joseph  Fox  states  in  Software  and  its  Development 
(1982:202)  that  flow  charts  are  no  longer  useful  as 
documentation.  He  goes  on  to  say  that  what  is  needed  is: 

1.  Wei  1 -Commented  Code  (SPS) 
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2.  Design  Diagrams  and  Narratives  (SSDD,  IDD,  SDD) 

3.  Structured  narratives  or  process  diagrams  (SDD) 

4.  Data  Descriptions  (SDD) 

Glass  and  Noiseux.  In  The  Software  Maintenance 
Guidebook ,  the  authors  maintain  that  any  documentation  that 
is  separate  from  the  code  will  not  be  maintained  and  should 
therefore  not  be  required  (1981:159).  They  do  consider  a 
top  level  overview  to  be  important  and  agree  that  it  should 
be  maintained  as  a  separate  document.  This  along  with  the 
source  code  would  constitute  the  SPS  as  the  only  document 
required  for  maintenance.  Code  that  contains  all  the 
information  needed  for  maintenance,  though,  would  be  so 
bulky  that  the  actual  code  may  become  lost  in  the  comments. 

Kempton  et  al .  In  this  article  by  UNISYS  Defense 
Systems  personnel,  the  authors  are  using  DOD-STD-2167A  as  a 
guide  and  list  the  following  as  required  for  software 
maintenance  (1988:162): 

1 .  SDP 

2  .  VDD 

3 .  STP 

4 .  STD 

5  .  STR 

6 .  SPS 

Fay  and  Holmes.  In  their  article  on  updating  an 
undocumented  program,  the  authors  (of  Lockheed  Aircraft 
Service  Company,  Software  Engineering  Department)  list  the 
following  documents  as  necessary  for  maintenance 
(1985:202) : 

1.  Source  Code  (SPS) 

2.  Description  Document  (SSDD,  IDD,  SDD) 
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3.  Users  Manual  (SUM,  CSOM) 

4.  Version  Description  Document  (VDD) 

5.  Test  Documentation  (STP,  STD,  STR) 

Martin  and  McClure.  In  their  book  Software 
Maintenance:  The  Problem  and  its  Solution,  the  authors  cite 
the  following  documents  as  necessary  for  software 
maintenance : 

1.  High  Level  Documentation  ( SRS ,  VDD,  SSDD) 

2.  User  Documentation  (SUM) 

3.  Operations  Documentation  (CSOM) 

4.  Source  Code  (SPS) 

5.  External  Program  Specification  (SRS) 

6.  Design  Documents  (SSDD,  IDD,  SDD) 

7.  System  and  Program  Flowcharts  (SSDD) 

8.  Cross-Sequence  Maps  (IDD) 

DOD-STD-2167A.  One  final  book  to  review  is  the  DOD 
standard  for  software  development.  The  standard  states 
that  the  following  five  documents  are  to  be  developed  and 
delivered  for  software  support  and  operation: 


1. 

CRISD 

2  . 

CSOM 

3  . 

SUM 

4  . 

SPM 

5  . 

FSM 

In  addition  to  this,  a  review  of  the  DIDs  shows  only 
three  described  as  possibly  being  used  for  maintenance. 
These  are  the  SPM,  the  FSM  and  the  SSDD. 


Anal ysis  . 
great  deal  of 
each  document 


As  can  be  seen  in  Table  1,  there  is  not  a 
agreement  on  what  documents  are  required, 
is  evaluated  on  the  basis  of  the  frequency 


If 
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TABLE  1 

Summary  of  Expert  Opinion 
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X 
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3 

2 

6 

B 

B 
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5 
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X 
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X 
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X 
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X 

X 
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X 
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X 
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X 

X 

DOD-STD-2167A 
and  DIDs 

X 

X 

X 

X 

X 

TOTALS 

4 

5 

5 

5 

2 

B 

0 

0 

with  which  the  experts  believe  the  document  is  necessary,  a 
tentative  prioritization  can  be  established. 

Three  of  the  documents  were  listed  in  seven  of  the 
references.  These  are  the  SSDD,  the  SDD,  and  the  SPS.  The 
IDD  was  the  only  document  referenced  six  times.  It  is 
reasonable  that  the  design  documents  would  be  required,  but 
the  SPS  actually  consists  of  two  documents:  the  source  code 
and  the  SDD,  Requiring  both  the  SDD  and  SPS  for 
maintenance  would  cause  two  problems.  First,  both 
documents  would  have  to  be  maintained,  causing  extra 
(unnecessary)  work  for  the  software  maintainers ,  and 
second,  having  two  separate  documents  that  (allegedly)  say 
the  same  thing  could  become  a  major  configuration  problem 
if  one  document  is  not  updated.  Since  the  SPS  contains 
both  the  SDD  and  the  source  code,  the  SPS  should  be 
supplied  for  maintenance  but  a  separate  SDD  should  not. 

Five  of  the  papers  called  for  the  VDD,  the  STP ,  the 
STR,  the  CSOM,  and  the  SUM;  four  cited  the  STD  as 
necessary;  three  agreed  that  the  SRS  and  CRISD  are  needed; 
and  only  two  believed  the  IRS  and  the  SPM  are  important. 
Only  one  report  stated  that  the  SDP  and  the  FSM  are 
necessary.  None  of  the  experts  referenced  the  information 
included  in  the  ECP ,  or  the  SON.  It  is  understandable  that 
the  ECP  and  SCN  were  not  mentioned,  since  the  information 
in  those  documents  is  added  to  other  documents  when  a 
change  is  performed. 
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Based  on  this  information.  Table  2  shows  a  tentative 


prioritization  of  the  documents. 

AFOTECP  800-2  Documentation  Questions 

AFOTECP  800-2  volume  3,  the  Software  Maintainability 
Evaluation  Guide,  is  used  by  the  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC)  to  rate  the  maintainability 
of  a  software  system.  If  the  evaluation  does  provide  an 
adequate  assessment  of  software  maintainability,  the 


TABLE  2 


Prioritization  Based  on  Frequency 
of  Experts  Opinions 


1. 

SSDD/SDD  or  SPS 

2  . 

IDD 

3. 

VDD/ STP/ STR/CSOM/ SUM 

4  . 

STD 

5. 

SRS/CRISD 

6  . 

IRS/SPM 

7. 

SDP/FSM 

information  the  evaluators  look  for  should  certainly 
provide  insight  into  what  documents  are  actually  required 
to  perform  maintenance.  The  guide  contains  68  questions 
related  to  the  documentation,  35  of  which  deal  directly 
with  the  content  of  different  documents  (the  remaining  33 
questions  deal  with  organization,  format,  traceability, 
etc.).  All  68  questions  are  listed  in  appendix  D.  Table  3 
lists  the  numbers  of  the  35  questions  that  relate  to 
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documentation  content  and  which  documents  from 
DOD-STD-2167A  are  addressed  by  each  question. 

Using  DOD-STD-2167A,  the  frequency  with  which  a 
document  was  referenced  can  influence  the  maintainability 
rating  gi.^n  by  AFOTEC .  If  all  questions  are  equally 
weighted,  then  the  documents  that  are  referenced  most  often 
will  have  a  greater  impact  on  the  maintainability  rating 
given  by  AFOTEC.  A  prioritization  based  on  the  frequency 
of  references  by  these  questions  is  given  in  Table  4. 

Table  3 

Relationship  between  AFOTECP  800-2  and  DOD-STD-2167A 


Question  Document(s) 


D-10 

IRS, 

IDD 

D-11 

VDD 

D-13 

SDD, 

SPS, 

SSDD 

D-14 

CSOM, 

SUM 

D-15 

CSOM, 

SUM 

D-16 

CSOM 

D-17 

CSOM 

D-19 

SDD, 

SPS 

D-20 

SDD, 

SPS 

D-21 

SDD, 

SPS, 

SSDD 

D-22 

SDD, 

SPS, 

SSDD 

D-23 

SDD, 

SPS 

D-24 

SDD, 

SPS 

D-25 

SDD, 

SPS 

D-26 

SRS 

D-27 

SDD, 

SPS 

D-28 

SDD, 

SPS, 

SPM 

D-29 

SDD, 

SPS 

Question  Document (s) 


D-30 

SSDD 

D-31 

SDD, 

SPS 

D-32 

SDD, 

SPS 

D-33 

SSDD 

D-34 

SSDD 

D-36 

SSDD 

D-37 

SSDD 

D-38 

SSDD 

D-39 

SDD 

D-40 

SPM 

D-48 

STP 

D-49 

STD 

D-50 

STD 

D-51 

STD 

D-52 

STD 

D-53 

STD 

D-54 

STD 
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Table  4 


Documents  Required  by  AFOTECP  800-2  volume  3 


1. 

SDD  or  SPS 

2. 

SSDD 

3  . 

STD 

4. 

CSOM 

5. 

SUM/SPM 

6. 

IDD/VDD/STP/SRS/ IRS 

Table  4  does  not  include  six  of  the  documents  listed  in 
DOD-STD-2167A.  As  stated  earlier,  the  ECP  and  SCN  should 
not  carry  over  into  the  maintenance  phase  and  their 
exclusion  from  this  list  was  expected.  Likewise,  the 
Software  Development  Plan  (SDP)  is  a  tool  used  to  track 
contractor  performance  and  although  it  may  be  used  by  the 
management  of  a  maintenance  organization  to  initially 
determine  the  scope  of  the  maintenance  effort,  it  is  not 
useful  in  the  actual  maintenance  of  the  system. 
Surprisingly,  the  need  for  the  Software  Test  Report  (STR) 
is  not  mentioned.  Its  main  purpose  is  to  document  previous 
tests,  but  it  also  includes  sections  about  problems 
encountered  and  recommendations  for  future  changes.  These 
sections  may  be  of  some  use  to  the  maintenance  programmer, 
and  the  STR  is  needed  for  regression  testing. 

The  CRISD  contains  information  about  the  support 
environment,  but  again,  this  information  is  probably  more 
useful  to  the  maintenance  organization  management  than  to 
the  actual  maintenance  programmer. 
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Finally,  the  FSM  is  not  listed  in  Table  4.  As  stated 
earlier,  if  the  system  includes  firmware,  the  FSM  is  an 
absolute  necessity. 

In  summary,  the  literature  review  was  not  conclusive. 
The  opinions  of  the  authors  varied  greatly  among  those 
reviewed  and  a  new  article  could  change  the  prioritization 
given.  The  analysis  of  AFOTECP  800-2  provides  a  better 
basis  for  documentation  requirements,  but  does  not  appear 
complete . 
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IV.  Survey 


Justification 

This  chapter  investigates  the  opinions  of  software 
maintainers  and  acquisition  personnel  within  the  Air  Force 
to  determine  which  documents  they  consider  necessary  for 
software  maintenance.  Because  of  the  number  of  responses 
required  and  the  amount  of  time  available,  a  mail  survey 
was  the  only  feasible  method  of  obtaining  the  required 
information , 

Survey  Instrument 

The  survey  (Appendix  A)  asks  five  questions  for  each  of 
the  18  documents  listed  in  DOD-STD-2167A.  The  first 
question  determines  the  availability  of  documents  to 
software  maintenance  personnel : 

Is  this  (or  a  similar)  document  available  for 

maintenance  of  your  software? 

a .  Yes 

b .  No 

Since  DOD-STD-2167A  is  relatively  new,  and  most 
software  in  existence  today  was  developed  under  old 
standards,  similar  documents  must  also  be  included  in  this 
investigation . 
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The  second  question  determines  the  requirement  for  each 
document  during  the  maintenance  phase  of  the  software 
life  cycl e : 

Without  this  document,  maintenance  is: 

a.  not  impacted 

b.  somewhat  difficult 

c.  difficult 

d.  very  difficult 

e.  impossible 

This  provides  a  range  of  choices  from  which  the 
respondent  can  choose  an  answer  that  shows  the  relative 
importance  of  the  document  without  necessarily  stating  that 
the  document  is  not  required  for  maintenance.  It  is 
generally  believed  that  if  a  document  is  not  absolutely 
required,  it  will  not  be  provided.  The  type  of  answer 
listed  above  will  provide  a  scaling  of  responses  to 
indicate  the  impact  of  not  having  a  specific  document. 

The  last  three  questions  address  tailoring  of  the  Data 
Item  Description  (DID)  for  a  particular  document.  Question 
three  asks: 

If  the  document  is  needed  for  maintenance,  was  the 

DID  tailored  to  meet  maintenance  requirements? 

a .  Yes 

b.  No,  tailoring  was  not  necessary 

c.  No,  but  it  should  have  been 

d .  Don ' t  know 

This  provides  general  information  about  tailoring  of 
the  DID. 
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Question  four  addresses  the  ALC's  participation  in  the 
tailoring  process: 

If  the  DID  was  tailored,  did  the  ALC  participate  in 

the  tailoring  process? 

a .  Yes 

b .  No 

c.  Don't  know 

Combined  with  question  five,  this  will  determine  the 
correlation  between  the  ALC's  participation  in  the 
tailoring  process  and  the  usability  of  the  document. 

Question  five  asks  about  the  amount  of  information 
present  in  a  document: 


If  the  DID  was  tailored,  how  did  the  amount  of 
information  required  for  software  maintenance 
compare  with  other  information  in  the  documents? 

a.  too  little  maintenance  information  and  an 
acceptable  amount  of  other  information. 

b.  an  acceptable  amount  of  both  maintenance 
information  and  other  information. 

c.  too  little  maintenance  information  and  too  much 
other  information. 

d.  an  acceptable  amount  of  maintenance  information 
and  too  much  other  information. 

This  provides  an  assessment  of  the  tailoring  done  on 


that  document  and  determines 
lost  in  reams  of  paperwork. 

Each  document  was  listed 
DID,  and  a  brief  description 
done  to  reduce  the  time 


if  the  needed  information  is 

survey  with  its  title, 
document.  This  was 
complete  the  survey  by 


in  the 
of  the 
required  to 


those  personnel  who  are  not  currently  using  DOD-3TD-2167A . 
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The  survey  also  asks  the  respondent  to  list  any  other 
documentation  (not  listed  in  DOD-STD-2167A)  that  is 
required  for  maintenance  and  what  type  of  organization  the 
respondent  works  for  (ALC  or  SPO) .  Space  is  also  provided 
for  comments. 

Population  to  be  Sampled 

As  stated  earlier,  two  different  populations  were 
sampled.  The  survey  is  limited  to  those  persons 
responsible  for  the  acquisition  and  maintenance  of  MCCR 
software  within  each  of  those  organizations.  The  ALCs 
provided  the  basis  for  determining  what  documentation  is 
required  as  well  as  what  is  currently  available.  The 
response  from  the  SPOs  gives  an  alternate  view  of  the 
documentation  requirements  and  may  indicate  whether 
training  is  required. 

The  number  of  people  directly  responsible  for  the 
maintenance  of  MCCR  software  is  estimated  to  be  5149  in 
AFLC  and  1497  in  AFSC  (AFLC,  1989).  To  determine  the 
sample  size  required  (for  a  90%  confidence  level),  the 
following  general  equation  applies  (HQ  USAF/ACM , 197 4 ) : 

2 

N( z  )p( 1  -  p) 

n  =  -  (1) 

2  2 

(N  -  l)(d  )  +  (z  )p(l  -  p) 
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where 

n  =  sample  size 
N  =  Population  size 

p  =  Maximum  sample  size  factor  (0.5) 
d  =  Desired  tolerance  (O.l) 

z  =  Factor  of  assurance  (1.645  for  90%  confidence 
1 evel ) 

This  results  in  a  sample  size  (n)  of  67  for  AFLC  and  65 
for  AFSC.  Assuming  a  low  return  rate,  360  surveys  were 
sent  --  200  to  AFLC  and  160 'to  AFSC.  The  return  rate  was 
not  consistent  between  the  commands,  with  103  responses 
(51%)  received  from  AFLC  and  only  27  responses  (17%) 
received  from  AFSC. 

Survey  Results 

A  complete  list  of  the  raw  survey  data  is  listed  in 
appendix  B.  Because  the  response  to  questions  related  to 
document  tailoring  was  insufficient  for  significant 
statistical  analysis,  this  paper  will  not  address  the  issue 
except  to  point  out  that  those  responses  received  indicate 
that  adequate  tailoring  is  being  performed.  This  is  not  to 
say  that  tailoring  is  not  needed,  but  generally  speaking, 
the  ALCs  are  happy  with  the  tailoring  that  has  been 
performed.  For  all  cases,  the  number  of  documents  that 
should  have  been  tailored,  but  were  not,  is  minimal 
compared  to  the  documents  thst  were  tailored  or  those  that 
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did  not  need  tailoring.  When  the  ALC  assisted  in  the 
tailoring  process,  the  'mount  of  maintenance  information 
and  other  information  was  well-balanced. 

Appendix  E  presents  the  tailoring  information  received 
in  the  form  of  a  table  for  each  document  that  compares  the 
amount  of  maintenance  information  and  other  information 
versus  the  ALC's  participation  in  the  tailoring  process. 
Table  5  presents  the  data  for  the  remaining  information  and 
lists  the  number  of  responses,  the  percent  avai  1  ab""  1  ity , 
and  the  mean  of  the  need  as  perceived  by  the  respondents, 
.ne  responses  for  the  need  were  converted  to  a  numerical 
value  where  A  =  1 ,  B  =  2 ,  C  =  3 ,  D  =  4 ,  and  E  =  5.  This 
implies  that  the  higher  the  need  rating,  the  more  crucial 
the  document  is  foi  software  maintenance. 

The  next  section  will  provide  an  analysis  of  this  data. 

Anal ysis 

This  section  reviews  the  results  of  the  survey  based  on 
the  data  presented  in  Table  5  in  order  of  mean  need  as 
perceived  by  the  ALC.  Each  document  is  presented 
separately  with  a  brief  description  and  comments  on 
availability  to  the  ALCs  and  the  perceived  need  of  both 
the  ALC  and  the  SPO .  The  analysis  is  based  on  the 
perceived  need  of  the  ALCs  since  they  are  the  ones  to 
actually  use  the  documents.  The  next  section  will  compare 
this  information  with  that  of  the  previous  chapter  and 
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Table  5. 
Survey  Data 


Number  of 

Percent 

Mean 

Need 

Responses 

Available 

(ALC) 

(  SPO) 

SPS 

129 

81.40 

3.939 

4.500 

SPM 

127 

72.44 

3.551 

2 . 526 

SDD 

126 

71.43 

3.305 

3 . 263 

FSM 

129 

48.06 

3.187 

2 . 526 

SRS 

126 

15. 40 

3.120 

2 . 889 

SUM 

129 

76.74 

3.081 

2.526 

IDD 

128 

54 . 69 

3.020 

3.333 

IRS 

127 

59.06 

2 . 970 

3.000 

CSOM 

128 

72 . 66 

2 . 928 

2.526 

SCN 

122 

69.67 

2 .844 

2 . 526 

ECP 

123 

73 . 17 

2 . 695 

2 . 526 

VDD 

112 

IB. 51 

2.656 

2.750 

STD 

128 

76.56 

2 .590 

2 . 526 

CRISD 

126 

58.73 

2.354 

2 . 526 

SSDD 

128 

53 . 91 

2.316 

2 . 278 

STP 

119 

85.71 

2.280 

2 . 421 

SDP 

129 

72.87 

1.938 

2 . 158 

STR 

128 

75.00 

1.860 

2.526 

finalize  the  documents  requirements  list.  The  documents 
are  discussed  in  order  of  their  presentation  in 
DOD-STD-2167A. 

System/ Segment  Design  Document.  The  SSDD  contains  the 
highest  level  design  information  for  the  system  or  segment. 
It  describes  the  organization  of  a  system  or  segment  as 
composed  of  Hardware  Configuration  Items  (HWCIs),  Computer 
Software  Configuration  Items  (CSCIs)  and  manual  operations 
( DI -CMAN-80534 ) .  The  DID  for  this  document  also  states 
that  the  SSDD  is  used  for  maintenance  of  the  system.  While 
this  document  appears  to  be  required  for  software 
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maintenance,  almost  half  of  the  respondents  do  not  have  the 
document.  This  may  be  the  cause  of  the  low  requirement 
rating  generated  by  the  ALCs  (2.316).  Those  that  have  the 
document  tended  to  give  it  a  higher  rating  that  those  that 
do  not . 

Software  Development  Plan.  The  SDP  describes  a 
contractor's  plans  for  conducting  software  development 
(DI -MCCR-80030A)  .  As  stated  in  chapter  3,  the  's  of 

little  use  in  the  actual  maintenance  of  a  software  system. 
This  accounts  for  the  low  rating  given  to  this  document 
(1.938)  by  the  ALCs.  Even  though  the  need  for  this 
document  is  virtually  nonexistent,  over  70  percent  of  the 
maintainers  have  the  SDP. 

Software  Requirements  Specification.  The  SRS  specifies 
the  engineering  and  qualification  requirements  for  a  CSCI 
and  is  the  basis  for  the  design  and  formal  testing  of  a 
CSCI  ( DI -MCCP-80025A) .  The  impact  of  not  having  this 
document  is  high  (3.120),  and  most  (75.40%)  of  the 
maintainers  have  access  to  it. 

Interface  Requirements  Specification.  The  IRS 
specifies  the  requirements  for  interfaces  between  one  or 
more  CSCIs  and  other  configuration  items  or  critical  items 
(DI-MCCR-80026A) .  Although  this  document  is  rated 
relatively  high  (2.970),  only  59  percent  of  the  maintainers 
have  access  to  it.  Those  that  have  the  IRS  rated  it  much 
higher  than  those  that  do  not.  The  lower  rating  may  have 
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resulted  from  systems  that  are  composed  of  a  single  CSCI 
and  therefore  do  not  require  interfaces. 

Interface  Design  Document.  The  IDD  specifies  the 
detailed  design  for  interfaces  between  one  or  more  CSCIs 
and  other  configuration  items  or  critical  items 
(DI-MCCR-80027A) .  Again,  this  document  is  rated  high 
(3.020),  though  few  (54.69%)  maintainers  have  it.  As  with 
the  IRS,  those  that  have  the  document  rate  it  higher  than 
those  that  do  not.  This,  again,  is  probably  due  to  system 
specific  needs . 

Software  Design  Document.  The  SDD  describes  the 
complete  design  of  a  CSCI  (DI-MCCR-80012A) .  This  is  a 
general  overview  of  the  entire  system.  The  SDD  received  a 
very  high  rating  (3.305)  and  is  available  to  most  (71.43%) 
of  the  maintainers.  See  analysis  for  the  SPS. 

Software  Product  Specification.  The  SPS  consists  of 
the  SDD  and  source  code  listings  for  a  CSCI 
(DI -MCCR-80029A) .  The  SPS  received  the  highest  rating  of 
all  the  documents  (3.939)  which  agrees  with  the  analysis  of 
chapter  3.  The  document  is  available  to  over  80  percent  of 
the  maintainers.  This  again  raises  the  question  of  whether 
this  document  and  the  SDD  should  both  be  maintained. 
According  to  the  survey  results,  both  are  rated  very  high 
(the  SPS  was  rated  first  and  the  SDD  was  third).  This  may 
be  because  of  tailoring  of  the  SPS.  That  is,  the  SPS  may 
only  include  references  to  the  SDD  without  actually 
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including  it,  but  no  data  is  available  to  substantiate 
this.  As  stated  in  chapter  3,  if  the  SPS  includes  the  SDD, 
both  documents  should  not  be  maintained  during  the  software 
maintenance  phase. 

Version  Description  Document.  The  VDD  identifies  and 
describes  a  version  of  a  CSCI  (DI-MCCR-80013A) .  This 
document  received  a  need  rating  of  2.656  and  is  available 
to  almost  80  percent  of  the  maintainers. 

Software  Test  Plan.  The  STP  describes  the  formal 
qualification  test  plans  for  one  or  more  CSCI 
( DI -MCCR-80014A) .  It  also  identifies  the  software  test 
environment  resources  required  for  Formal  Qualification 
Testing  (FQT)  and  identifies  individual  tests  performed 
during  FQT.  Although  this  document  received  a  relatively 
low  rating  by  the  ALCs  (2.280),  it  was  ranked  very  high  in 
chapter  3.  Compared  to  the  other  test  documents,  though, 
the  STP  is  more  of  a  management  tool  than  a  tool  required 
by  the  maintainer. 

Software  Test  Description.  The  STD  contains  the  test 
cases  and  test  procedures  necessary  to  perform  formal 
qualification  testing  of  a  CSCI  identified  in  the  STP 
( DI -MCCR-80015A) .  Of  all  the  test  documents,  the  STD 
received  the  highest  rating  (2.590)  and  is  available  to 
most  of  the  maintainers  (76.56%).  Since  this  document 
contains  the  actual  test  cases,  it  is  of  more  immediate  use 
to  the  maintainers  than  the  other  test  documents. 
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Software  Test  Report.  The  STR  is  a  record  of  the 


formal  qualification  testing  performed  on  a  CSCI 
( DI -MCCR-80017A) .  This  document  received  the  lowest  rating 
of  all  documents  on  the  survey  (1.860)  but  is  available  to 
fully  three  quarters  of  the  maintainers.  As  stated  in 
chapter  3,  the  STR  does  have  some  maintenance  related 
information.  Specifically,  deviations  from  the  test 
procedure  should  be  considered,  recommended  improvements 
should  be  reviewed  by  the  maintenance  management,  and  the 
report  is  needed  as  a  baseline  for  regression  testing. 

Computer  System  Operator’s  Manual.  The  CSOM  provides 
information  and  detailed  procedures  for  initiating, 
operating,  monitoring,  and  shutting  down  a  computer  system 
and  for  identifying  or  isolating  a  malfunctioning  component 
in  a  computer  system  ( DI -MCCR-80018A) .  Compared  to  the 
other  documents,  the  CSOM  received  a  relatively  high  rating 
(2.928)  and  is  available  to  over  70  percent  of  the  software 
maintainers.  Those  maintainers  that  don't  have  the  CSOM 
rated  it  significantly  lower  than  those  who  have  it.  This 
is  probably  due  to  the  fact  that  the  information  can  be 
found  in  Technical  Orders  (TOs)  for  many  onboard  systems. 
Although  the  survey  did  not  address  TOs  or  associate  TOs 
with  the  CSOM  or  SUM,  the  responses  to  the  CSOM  and  the  SUM 
(see  below)  indicate  that  user's  manuals  are  very  important 


35 


to  maintenance  whether  they  are  in  the  form  of  a  CSOM,  SUM, 
TOs ,  or  Commercial  Off  The  Shelf  (COTS)  manuals.  This  was 
also  obvious  from  the  comments  received  (see  Appendix  C) . 

Software  User's  Manual.  The  SUM  provides  user 
personnel  with  instructions  sufficient  to  execute  one  or 
more  related  CSCIs  ( DI -MCCR-80019A) .  This  document 
received  a  high  rating  (3.081)  and  is  available  to  most  of 
the  maintainers.  See  the  analysis  of  the  CSOM  for  more 
detai 1 s . 

Software  Programmer’s  Manual.  The  SPM  provides 
information  needed  by  a  programmer  to  understand  the 
instruction  set  architecture  of  the  specified  host  and 
target  computers  ( DI -MCCR-80021A) .  This  document  is 
important  for  the  maintainance  of  any  software  system  and 
received  the  second  highest  rating  on  the  survey  (3.551). 

It  is  available  to  over  70  percent  of  the  maintainers. 

Firmware  Support  Manual.  The  FSM  provides  the 
information  necessary  to  load  software  or  data  into 
firmware  components  of  a  system  ( DI -MCCR-80022A ) .  As 
stated  in  chapter  3,  if  firmware  exists  in  a  system,  then 
the  FSM  is  an  absolute  necessity  for  maintenance. 
Unfortunately,  firmware  is  often  viewed  as  hardware  and 
considered  permanent  in  its  original  configuration.  This 
document  received  a  very  high  rating  (3.187),  but  less  than 
50  percent  of  the  maintainers  have  the  document.  Although 
no  data  was  collected  concerning  the  existence  of  firmware. 
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those  who  do  not  have  the  FSM  rated  it  approximately  2.75, 
which  indicates  that  it  is  not  always  present  when  needed. 

Computer  Resources  Integrated  Support  Document.  The 
CRISD  provides  the  information  needed  to  plan  for  lifecycle 
support  of  deliverable  software  and  is  used  for  updating 
the  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP) 
(DI-MCCR-80024A) .  This  document  received  a  relatively  low 
rating  (2.354),  and  only  58  percent  of  the  maintainers  have 
it.  Since  the  CRLCMP  must  be  maintained  by  the  maintenance 
organization,  maintaining  the  same  information  in  the  CRISD 
is  not  only  unnecessary  but  unwise. 

Engineering  Change  Proposal .  The  ECP  includes  both  a 
proposed  engineering  change  and  the  documentation  by  which 
the  change  is  described  and  suggested  (DI-CMAN-80639) . 

This  document  received  a  rating  of  2.695  which  is  higher 
than  originally  expected.  As  stated  in  chapter  3,  once  an 
ECP  IS  approved,  other  documents  are  modified  to  include 
the  changes,  so  ECPs  written  during  design  or  development 
should  be  of  little  use  during  maintenance.  The  unexpected 
high  rating  may  be  due  to  the  requirement  to  generate  new 
ECPs  during  the  software  maintenance  phase.  The  only  ECPs 
written  during  development  that  may  be  of  interest  would  be 
those  that  were  disapproved.  If  the  decision  makers  in 
charge  of  the  maintenance  organization  have  the  disapproved 
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ECPs  with  the  reason  each  was  not  incorporated,  they  will 
not  spend  tirpe  re-evaluating  similar  proposals  that  are 
submitted  during  the  maintenance  phase. 

Specification  Change  Notice.  The  SCN  is  used  to 
delineate  the  exact  change{s)  in  a  specification  that  will 
be  distributed  to  users  when  the  SCN  is  approved 
(DI -CMAN-80643 ) .  Again,  the  rating  of  2.844  is 
unexpectedly  high,  but  probably  for  the  same  reason  as  the 
ECP .  SCNs  might  be  developed  during  the  maintenance  phase 
prior  to  their  incorporation  into  the  required 
specification.  The  SCNs  produced  during  the  development 
phase  should  be  of  little  interest  during  the  maintenance 
phase . 

Conclusion 

The  current  availability  of  documentation  does  not 
correspond  to  the  need  for  those  documents.  Documentation 
that  was  needed  for  the  development  phase,  but  not  useful 
during  software  maintenance,  is  readily  available; 
documents  not  needed  to  track  development,  but  required  tor 
maintenance,  are  sometimes  scarce.  This  section  divides 
the  documentation  into  three  categories  that  describe  the 
need  during  the  maintenance  phase  of  the  software 
life  cycle.  These  three  categories  are: 

1.  Documentation  needed  by  the  maintenance 
programmer . 
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2.  Documentation  needed  by  the  management  of  the 
maintenance  organization, 

3.  Documentation  not  needed  during  the  maintenance 
phase  of  the  software  life  cycle. 

Note  that  the  third  category  does  not  imply  that  the 

maintenancia  organization  should  not  be  included  in  the 

tailoring  or  review  of  these  documents  during  the 

development  phase,  but  that  when  the  system  is  delivered, 

these  documents  are  only  of  historical  interest. 

The  following  paragraphs  provide  guidance  for  delivery 
of  final  documentation  to  the  support  organization  but 
should  not  be  interpreted  as  the  final  word  on  the  matter. 
They  provide  a  minimum  requirement  for  software 
maintenance.  The  support  organization  should  participate 
fully  in  tailoring  the  DIDs  as  well  as  spelling  out 
precisely  which  documents  are  required  during  the 
maintenance  phase.  The  documentation  requirements  for 
software  maintenance  should  be  included  in  both  the  CRLCMP 
and  the  PMRT  agreement.  The  support  organization  should 
also  be  included  in  the  review  of  preliminary  documents  and 
approval  of  those  documents  needed  for  maintenance. 

Documentation  Needed  by  the  Maintenance  Programmer. 

The  documentation  needed  for  maintenance  of  a  software 
systGiTi  can  be  divided  into  four  categories! 

1.  General  overview  of  the  system. 

2.  Specifics  of  the  system. 

3.  Current  implementation  of  the  system. 

4.  Testing  of  the  system. 
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The  general  overview  of  the  system  includes  the  SPS, 
the  SRS,  the  SPM,  and  the  FSM.  Although  the  SPM  and  FSM 
could  be  considered  specifics  of  the  system,  they  are 
included  here  because  they  represent  programming  practices 
for  the  entire  system. 

The  specifics  of  the  system  are  documented  in  the  IDD, 
the  IRS,  and  the  SSDD. 

The  current  implementation  of  the  system  is  described 
in  the  SUM,  the  CSOM,  and  the  VDD.  Applicable  TOs  and 
commercial  manuals  should  also  be  included  in  this 
category . 

Testing  of  the  system  is  covered  by  the  STD. 

In  summary.  Table  6  shows  what  documents  are  required 
by  the  maintenance  programmer  and  lists  them  in  order  of 
the  mean  of  the  need  as  perceived  by  the  ALCs . 

Table  6 


Documents 

Requi red 

by  Software 

Maintainers 

1 . 

SPS 

5. 

SUM 

9.  VDD 

2  . 

SPM 

6. 

IDD 

10.  STD 

3. 

FSM 

7  . 

IRS 

11.  SSDD 

4  . 

SRS 

8  . 

CSOM 

Note  : 

Any  other  documentation  that  explains  the  use  of  the 

system 

(TOs,  COTS 

manua 1 s , 

etc . )  should 

also  be  delivered. 

Documentation  Needed  by  the  Management  of  the 
Maintenance  Organization.  The  following  documentation 
should  be  provided  to  the  management  of  the  maintenance 
organization  for  initial  planning  of  the  maintenance  phase 
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of  the  system  life  cycle.  These  documents  appear  to  be  of 
limited  use  to  the  actual  maintenance  programmer. 

1 .  SDP 

2 .  STP 

3.  Disapproved  ECPs 

4 .  STR 

The  management  should  also  have  the  CRLCMP,  but  this 
document  generally  becomes  the  responsibility  of  the 
maintenance  organization  when  the  software  is  turned  over 
to  them. 

Documentation  Not  Needed  During  the  Maintenance  Pha s e 
of  the  Software  Life  Cycle.  The  following  documents  are 
not  needed  during  the  maintenance  phase  of  the  software 
system,  and  to  reduce  the  maintenance  burden,  should  not 
be  maintained  after  the  system  is  developed. 

1  .  SDD 

2  .  SCN 

3.  CRISD 

4.  Incorporated  ECPs 

The  SDD  is  not  needed  because  it  should  become  part  of 
the  SPS .  The  SCNs  are  preliminary  documents  with  their 
content  added  to  the  respective  specification  on  approval . 
Information  needed  from  the  CRISD  should  be  incorporated 
into  the  CRLCMP.  ECPs  that  were  approved  and  incorporated 
should  be  reflected  in  the  appropriate  design  documents  and 
specifications . 

Comparison  of  Survey  Results  with  Existing  Information . 
All  of  the  documents  listed  in  Table  6  are  included  in 
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Table  2  (Prioritization  based  on  frequency  of  expert 
opinions)  and  all  except  the  FSM  are  in  Table  4  (Documents 
required  by  AFOTECP  800-2).  As  explained  in  chapter  3,  if 
firmware  exists  then  the  FSM  is  required  and  AFOTEC  may 
have  other  procedures  to  deal  with  firmware.  Of  those 
documents  required  by  the  management  of  a  maintenance 
organization,  only  the  disapproved  ECPs  are  missing  from 
Table  2;  Table  4  mentions  only  one  of  those  documents  (the 
STP).  The  documents  listed  as  not  needed  during  the 
maintenance  phase,  with  the  exception  of  the  SDD ,  are  not 
present  in  any  of  the  relevant  tables.  As  stated  earlier, 
the  SDD  should  not  exist  concurrently  with  the  SPS . 

In  summary,  Table  6  provides  a  more  complete  basis 
for  documentation  requirements,  but  as  stated  earlier, 
should  be  used  as  a  minimum  requirement,  not  a  complete 
list.  Other  documents  should  be  evaluated  independently 
based  on  specific  system  requirements. 
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V.  Conclusion 


Significance  of  Research 

This  research  has  shown  that  the  Air  Force  often 
provides  insufficient  or  unnecessary  information  to  the 
software  maintenance  organizations.  The  information  most 
needed  is  often  not  available,  while  documents  that  are  not 
useful  during  software  maintenance  are  provided.  This 
results  in  software  maintainers  booking  through  enormous 
piles  of  paperwork  trying  to  understand  the  software,  while 
the  needed  information  is  often  not  there.  With  this 
situation,  the  only  way  to  understand  the  software  is  to 
reverse  engineer  the  source  code  and  attempt  to  recreate 
the  thought  process  of  the  original  designers.  If  the 
documents  listed  in  Table  6  are  properly  developed  and 
provided  to  the  software  maintainers,  their  ability  to 
understand  the  system  will  be  greatly  enhanced,  thus 
reducing  a  large  portion  of  the  maintenance  cost. 
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system,  additional  documerJ  ^tion  may  be  required  (for 
example,  maintenance  of  ATL  software  requires  significant 
information  about  the  Unit  Under  Test).  To  effectively  use 
the  information  provided  in  this  thesis,  the  acquisition 
agency  and  the  maintenance  organization  must  work  closely 
together  throughout  software  acquisition. 

The  maintenance  organization  must  be  consulted  early  in 
the  software  life  cycle  to  determine  exactly  which 
documents  are  needed,  and  this  information  should  be 
documented  in  initial  PMRT  agreements  and  in  the  CRLCMP. 
While  the  maintenance  documents  are  being  developed,  the 
maintenance  organization  must  be  allowed  sufficient  time  to 
review  the  content  to  ensure  that  the  information  being 
provided  is  adequate.  The  maintenance  organization  should 
have  approval  authority  for  any  documents  that  are  used 
only  for  maintenance.  Additionally,  some  way  must  be  found 
to  deliver  the  proper  final  documents  to  the  maintenance 
organization.  During  the  acquisition  nrocess ,  the 
maintenance  organization  should  be  included  in  the  review 
of  all  documentation  submitted  by  the  contractor,  but  when 
the  system  is  delivered,  only  the  final  documents  that  are 
needed  for  maintenance  should  be  delivered. 

R  e  c  pmmenda  t i on  s 

This  section  makes  recomir  endat  i  ons  needed  both  to 
implement  the  findings  of  this  thesis  and  for  additional 
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research  that  may  further  enhance  the  software  maintenance 
effort. 

Revision  of  Data  Item  Descriptions  (DIDs).  As  stated 
in  chapter  3,  only  three  of  the  DIDs  called  for  by 
DOD-STD-2167A  state  that  the  document  may  be  used  for 
maintenance.  The  DIDs  related  to  the  documents  listed  in 
Table  6  should  include  a  statement  in  block  3 
(Description/purpose)  that  says  the  document  is  required 
for  software  maintenance.  The  DIDs  for  the  documents 
required  for  management  activity  should  state  in  block  3 
that  the  document  is  required  by  the  management  of  the 
maintenance  organization  to  initially  set  up  the  software 
maintenance  capability. 

Content  of  the  Maintenance  Documents.  During  the 
course  of  this  research,  it  was  found  that  some  information 
is  repeated  in  different  documents  (for  example,  the  IRS 
and  the  IDD  contain  many  identical  sections).  This  adds  to 
the  maintenance  burden  by  requiring  that  multiple  documents 
be  updated  during  a  maintenance  action.  Further  research 
IS  needed  to  determine  if  the  format  prescribed  by  the 
combined  DIDs  communicate  the  ideas  of  the  software 
designers  in  the  best  possible  manner. 

Training  of  Personnel .  Comments  received  with  the 
survey  responses  show  a  lack  of  understanding  by  both 
acquisition  and  maintenance  personnel  of  the  content  of  the 
documents  addressed.  Many  comments  concerning  additional 
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documentation  needed  listed  the  same  documents  that  were 
asked  about  or  information  that  is  contained  in  the 
DOD-STD-2167A  documents.  Comments  from  the  SPOs  often 
stated  that  the  respondents  were  unfamiliar  with  the 
documents  or  the  needs  of  the  maintenance  organization. 

This  type  of  training  is  available,  as  a  short  course, 
through  AFIT/EN  (WCSE  474,  Software  Generation  and 
Maintenance)  and  will  familiarize  personnel  with  the 
importance  and  requirements  of  the  maintenance  phase  of  the 
software  life  cycle. 

Tailoring  of  the  Maintenance  Documents.  Although  the 
survey  responses  generally  indicated  that  tailoring  was 
adequate,  the  response  to  questions  on  tailoring  was 
limited  and  may  not  be  statistically  significant.  If  the 
documentation  is  not  properly  tailored,  the  delivered 
documents  could  contain  too  much  unnecessary  information  or 
lack  crucial  information.  Although  inclusion  of  the 
maintenance  organization  in  the  tailoring  process  should 
ensure  all  necessary  information  is  present,  the 
maintenance  organization  is  often  not  identified  until 
after  a  new  development  is  placed  on  contract.  Research 
should  be  performed  that  will  provide  a  tailoring  plan  for 
the  maintenance  documents  that  can  be  used  by  the 
acquisition  organization  to  ensure  maintenance  requirements 
are  adequately  addressed. 
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Appendix  A:  Survey 

Software  Documentation  Requirements  Survey 


Survey  Control  Number:  91-12 
Expiration  Date:  1  August  91 

DOD-STD-2167A  lists  18  documents  that  should  be  developed 
during  the  acquisition  of  a  software  system.  On  any  major 
software  system,  this  becomes  an  enormous  amount  of  paperwork  for 
any  organization  to  understand  and  maintain.  We  would  like  to 
know  which  documents  the  ALCs  use  most  often  in  maintaining 
software,  and  which  documents  the  SPOs  believe  are  necessary  for 
software  maintenance. 

This  survey  is  being  sent  to  the  Air  Logistic  Centers  (ALCs) 
and  System  Program  Offices  (SPOs)  in  order  to  obtain  different 
perspectives  on  the  subject.  Please  answer  the  questions  based 
on  the  perspective  of  your  organization  and  the  software  for 
which  you  are  responsible. 

For  the  purpose  of  this  survey,  software  maintenance  will  be 
defined  as  the  modification  of  a  software  product  after  delivery 
to  correct  faults,  improve  performance,  or  to  adapt  the  software 
to  a  new  environment. 

Since  DOD-STD-2167A  is  relatively  new,  and  most  systems  were 
developed  using  other  standards,  a  brief  description  of  each 
document  is  listed.  If  a  document  similar  to  the  one  listed  is 
available,  it  should  be  used  as  a  basis  for  completing  the 
survey . 

Additional  comments,  explanations,  and  suggestions  are 
welcome  and  will  be  considered  in  the  final  report.  Address  all 
questions  and  comments  to  Capt  Timothy  McArthur,  AFIT/LSG, 
Wright-Patterson  AFB  OH  45433  (Defense  Switched  Network 
785-8989)  . 


Completing  the  Survey 

This  survey  contains  five  questions  for  each  of  the  18 
documents  listed  in  DOD-STD-2167A.  To  save  room,  the  complete 
questions  are  listed  on  page  2,  and  an  abbreviated  versi..<n  of  the 
questions  is  listed  on  each  page.  Each  document  is  then  listed 
with  the  applicable  Data  Item  Description  (DID)  number  and  a 
brief  description  of  the  document.  The  right  side  of  the  survey 
provides  space  to  answer  the  questions  prior  to  filling  in  the 
computer  answer  sheet,  if  desired.  The  first  two  questions 
should  be  answered  for  each  document,  the  third  question  should 
be  answered  only  if  the  document  is  required  for  maintenance,  and 
is  available.  The  last  two  questions  should  be  answered  only  if 
tailoring  of  the  DID  was  performed. 
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Software  Documentation  Requirements  Survey 


Complete  questions 

These  five  questions  will  be  asked  for  each  of  the  18 
documents  listed  in  DOD-STD-2167A .  Because  of  the  length 
of  the  questions,  the  complete  questions  are  listed  here 
and  an  abbreviated  version  is  listed  on  each  page  of  the 
survey . 

I.  Is  this  (or  a  similar)  document  available  for 
maintenance  of  your  software? 

a .  Yes . 

b .  No . 

II.  Without  this  document,  maintenance  is: 

a.  not  impacted 

b.  somewhat  difficult 

c.  difficult 

d.  very  difficult 

e.  impossible 

III.  If  the  document  is  needed  for  maintenance,  was  the 
DID  tailored  to  meet  maintenance  requirements? 

a.  Yes. 

b.  No,  tailoring  was  not  necessary. 

c.  No,  but  it  should  have  been. 

d.  Don't  know. 

IV.  If  the  DID  was  tailored,  did  the  ALC  participate 
in  the  tailoring  process? 

a .  Yes . 

b.  No . 

c.  Don't  know. 

V.  If  the  DID  was  tailored,  how  did  the  amount  of 
information  required  for  software  maintenance  compare 
with  the  amount  of  other  information  in  the  document? 

a.  too  little  maintenance  information  and  an 
acceptable  amount  of  other  information. 

b.  an  acceptable  amount  of  both  maintenance 
information  and  other  information. 

c.  too  little  maintenance  information  and  too 
much  other  information. 

d.  an  acceptable  amount  of  maintenance  information, 
and  too  much  other  information. 


Software  Documentation  Requirements  Survey 


Abbreviated 

Questions 

(for  complete 

question 

,  see  page  2) 

I . 

Document  is: 

IV. 

ALC  help  tailor  DID 

a . 

avai 1 abl e 

a . 

did 

b. 

not  available 

b. 

did  not 

c . 

don't  know 

II . 

Without  document. 

maintenance  is: 

V. 

Document  contained: 

a . 

not  impacted 

a . 

too  little  maintenance 

b. 

somewhat  difficult 

information 

c . 

dif f icul t 

b. 

acceptable  amount  of 

d. 

very  difficult 

maintenance  information 

e . 

impossibl e 

c . 

too  little  maintenance 

inf ormation/ too  much 

Ill 

.  Tailoring  was: 

other  information 

a . 

performed 

d. 

acceptable  amt  of 

b. 

not  performed/not  needed 

maintenance  information/ 

c . 

not  performed/needed 

too  much  other 

d. 

don't  know 

information 

Question  #  I  II  III  IV  V 


Software  Documentation  Requirements  Survey 


the  DID:  4  A  B'  C  D  E 

If  the  information  needed  for 
software  maintenance  is  available, 
but  lost  in  a  lot  of  needless 

information:  5  A  B  C  D  E 
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Software  Documentation  Requirements  Survey 


Abbreviated  Questions 
(for  complete  question,  see  page  2) 


Document 


help  tailor  DID 


a.  available 

a . 

did 

b.  not  available 

b. 

did  not 

Without  document, 
maintenance  is: 

V. 

c.  don't  know 

Document  contained: 

a.  not  impacted 

a . 

too  little  maintenance 

b.  somewhat  difficult 

c.  difficult 

b. 

information 
acceptable  amount  of 

d.  very  difficult 

e.  impossible 

c . 

maintenance  information 
too  little  maintenance 

Tailoring  was: 
a.  performed 

d. 

inf ormation/too  much 
other  information 
acceptable  amt  of 

b.  not  performed/not  needed 

c.  not  performed/needed 

d.  don't  know 


Question  No 


System/Segment  Design  Document 
DI-CMAN-80534 

Describes  the  organization  of  a 
system  or  segment  as  composed  of 
Hardware  Configuration  Items  (HWCIs), 
Computer  Software  Configuration  Items 
(CSCIs)  and  manual  operations. 


Software  Development  Plan 
DT-MCCR-80030A 

Describes  the  contractor's  plans  for 
conducting  software  development. 


Software  Requirements  Specification 
DI-MCCR-80025A 

Specifies  the  engineering  and 
qualification  requirements  for  a 
CSCI .  Specifies  the  requirements 
allocated  to  a  CSCI. 


maintenance  information/ 
too  much  other 
information 
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Software  Documentation  Requirements  Survey 


Abbreviated 

Questions  11 

(for  complete 

question , 

see  page  2) 

I . 

Document  is: 

IV. 

ALC  help  tailor  DID 

a . 

avai 1  able 

a . 

did 

b. 

not  available 

b. 

did  not 

c . 

don't  know 

1 1 . 

Without  document. 

maintenance  is: 

V . 

Document  contained: 

a . 

not  impacted 

a . 

too  little  maintenance 

b. 

somewhat  difficult 

information 

c . 

dif f icul t 

b. 

acceptable  amount  of 

d. 

very  difficult 

maintenance  information 

e . 

impossible 

c . 

too  little  maintenance 

inf ormation/too  much 

Ill 

Tailoring  was: 

other  information 

a . 

performed 

d. 

acceptable  amt  of 

b. 

not  performed/not  needed 

maintenance  information/ 

c . 

not  performed/needed 

too  much  other 

d. 

don ' t  know 

information 

Question 

No. 

I  II  III  IV  V 

Interface  Requirements  Specification 

16. 

17  . 

00 

19. 

20. 

DI-MCCR-80026A 

a 

a 

a 

a 

a 

Specifies  the  requirements  for  one  or 

b 

b 

b 

b 

b 

more  interfaces  between  one  or  more 

c 

c 

c 

CSCIs  and  other  configuration  items 

d 

d 

or  critical  items. 

e 

Interface  Design  Document 

21. 

22  . 

23. 

24. 

25  . 

DI-MCCR-80027A 

a 

a 

a 

a 

a 

Specifies  the  detailed  design  for 

b 

b 

b 

b 

b 

one  or  more  interfaces  between  one 

c 

c 

c 

or  more  CSCI(s)  and  other 

d 

d 

configuration  items  or  critical 

e 

i t  ems . 

Software  Design  Document 

26. 

27  . 

28  . 

29  . 

30  . 

DI-MCCR-80012A 

a 

a 

a 

a 

a 

Describes  the  complete  design  of  a 

b 

b 

b 

b 

b 

CSCI.  It  breaks  the  CSCI  into 

c 

c 

c 

Computer  Software  Components  (CSCs) 

d 

d 

and  Computer  Software  Units  (CSUs). 

e 
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Abbreviated  Questions 
(for  complete  question,  see  page  2) 


Document 


help  tailor  DID 


a 

.  available 

a . 

did 

b 

.  not  available 

b. 

did  not 

c . 

don't  know 

Without  document, 

maintenance  is: 

V. 

Document  contained: 

a 

not  impacted 

a . 

too  little  maintenance 

b 

somewhat  difficult 

information 

c 

di f  f icul t 

b. 

acceptable  amount  of 

d 

very  difficult 

maintenance  information 

e 

impossibl  e 

c . 

too  little  maintenance 

inf ormation/too  much 

Tailoring  was: 

other  information 

a . 

,  performed 

d. 

acceptable  amt  of 

b.  not  performed/not  needed 

c.  not  performed/needed 

d.  don't  know 


maintenance  information/ 
too  much  other 
information 


Question  No.  I 


Software  Product  Specification 
DI-MCCR>80029A 

Consists  of  the  Software  Design 
Document  (SDD)  and  source  code 
listings  for  a  CSCI . 


Version  Description  Document 
DI-MCCR'80013A 
Identifies  and  describes  a  v 
of  a  CSCI . 


version 


Software  Test  Plan 
DI-MCCR-80014A 

Describes  the  formal  qualification 
test  plans  for  one  or  more  CSCI. 
Identifies  the  individual  tests  that 
shall  be  performed  during  Formal 
Qualification  Testing. 
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Software  Documentation  Requirements  Survey 


Abbreviated  Questions 
(for  complete  question,  see  page  2) 


Document  is: 

IV. 

ALC  helo  tailor  DID 

a . 

avai 1  able 

a . 

did 

b. 

not  available 

b. 

did  not 

c . 

don't  know 

Without  document. 

maintenance  is: 

V. 

Document  contained: 

a . 

not  impacted 

a . 

too  little  maintenance 

b. 

somewhat  difficult 

information 

c . 

difficult 

b. 

acceptable  amount  of 

d. 

very  difficult 

maintenance  information 

e . 

impossibl e 

c . 

too  little  maintenance 

inf ormation/too  much 

Tailoring  was: 

other  information 

a . 

performed 

d. 

acceptable  amt  of 

b. 

not  performed/not  needed 

maintenance  information/ 

c . 

not  performed/needed 

too  much  other 

d. 

don ' t  know 

information 

Question  No.  I 

II 

III 

IV 

- •• 

V 

Software  Test  Description 

46. 

47. 

48. 

49. 

50  . 

DI-MCCR-80015A 

a 

a 

a 

a 

a 

Contains  the  test  cases  and  test 

b 

b 

b 

b 

b 

procedures  necessary  to  perform 

c 

c 

c 

Formal  Qualification  Testing  (FQT) 

d 

d 

of  a  CSCI  identified  in  the  software 

e 

test  plan. 

Software  Test  Report 

51. 

52. 

53. 

54. 

55. 

DI-MCCR-80017A 

a 

a 

a 

a 

a 

A  record  of  the  formal  FQT  performed 

b 

b 

b 

b 

b 

on  a  CSCI. 

c 

c 

c 

d 

e 

d 

Computer  System  Operator's  Manual 

56. 

57  . 

58  . 

59. 

60  . 

DI-MCCR-80018A 

a 

a 

a 

a 

a 

Provides  information  and  detailed 

b 

b 

b 

b 

b 

procedures  for  initiating,  operating. 

c 

c 

c 

monitoring,  and  shutting  down  a 

d 

d 

computer  system  and  for  identifying/ 

e 

isolating  problems. 
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Abbreviated  Questions 
(for  complete  question,  see  page  2) 


Document 


help  tailor  DID 


a.  available 

a . 

did 

b.  not  available 

b. 

did  not 

Without  document. 

c . 

don ' t  know 

maintenance  is: 

V.  Document  contained: 

a.  not  impacted 

a . 

too  little  maintenance 

b.  somewhat  difficult 

information 

c.  difficult 

b. 

acceptable  amount  of 

d.  very  difficult 

maintenance  information 

e.  impossible 

c . 

too  little  maintenance 
inf ormation/too  much 

Tailoring  was: 

other  information 

a.  performed 

d. 

acceptable  amt  of 

b.  not  performed/not  needed 

c.  not  performed/needed 

d.  don't  know 


maintenance  information/ 
too  much  other 
information 


Question  No.  I 


Software  User's  Manual 
DI-MCCR-80019A 
Provides  user  personnel  with 
instructions  sufficient  to  execute 
one  or  more  related  CSCIs.  Provides 
steps  for  executing  the  software, 
expected  output,  and  errors. 


Software  Programmer's  Manual 
DI-MCCR-80021A 

Provides  information  needed  by  a 
programmer  to  understand  the 
instruction  set  architecture  of  the 
specified  host  and  target  computers. 


Firmware  Support  Manual 
DI-MCCR-80022A 

Provides  the  information  necessary 
to  load  software  or  data  into 
firmware  components  of  a  system. 
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Abbreviated  Questions 


(for  complete 

I.  Document  is: 

a.  available 

b.  not  available 

II.  Without  document, 
maintenance  is: 

a.  not  impacted 

b.  somewhat  difficult 

c.  difficult 

d.  very  difficult 

e.  impossible 

III.  Tailoring  was: 

a. ‘  performed 

b.  not  performed/not  needed 

c.  not  performed/needed 

d.  don’t  know 


Question 


question,  see  page  2 

IV.  ALC  _  help  tailor  DID 

a .  did 

b.  did  not 

c.  don't  know 

V.  Document  contained: 

a.  too  little  maintenance 
information 

b.  acceptable  amount  of 
maintenance  information 

c.  too  little  maintenance 
inf ormation/too  much 
other  information 

d.  acceptable  amt  of 
maintenance  information/ 
too  much  other 
information 

No.  I  II  III  IV  V 


Computer  Resources  Integrated  Support 
Document  -  80024A 

Provides  the  information  needed  to 
plan  for  the  life  cycle  support  of 
deliverable  software. 


Engineering  Change  Proposal 
DI-CMAN-80639 

Provides  a  complete  analysis  of  the 
technical,  interface,  cost,  schedule 
and  logistics  impacts  of  a  proposed 
change . 


Specification  Change  Notice 
DI-CMAN-80643 

Delineates  the  exact  change(s) 
a  specification  that  will  be 
distributed  to  users  when  the 
Specification  Change  Notice  is 
approved . 
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91.  Are  other  documents  required  for  software  maintenance  that 
are  not  listed  above? 

a.  Yes  (please  list) 

b.  No 

92.  For  which  type  of  organization  do  you  work? 

a .  ALC 

b.  SPO 

93.  Was  the  documentation  that  you  use  developed  using 
EOD-STD-2167A? 

a .  Yes 

b .  No 


Optional  information 

Name:  _ 

Organization:  _ 

Title:  _ 

Phone:  _ 

Additional  Comments: 
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Appendix  B:  Survey  Responses 


Response  Question 

No.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17 


1  B  C 

2  A  B 

3 

4  A  A 
SAC 

6  A  A 

7  B  A 

8  B  D 

9  B 

10  B  A 

11  A  A 

12  B  A 

13  B  C 

14  B  D 

15  B  C 

16  B  A 

17  B  A 

18  A  B 

19  B  D 

20  A  E 

21  A  B 

22  B  A 

23  A  A 

24  B  A 

25  B  D 

26  A  A 

27  A  A 

28 

29  B  D 

30  A  A 

31  A  B 

32  A 

33  A  B 

34  B  A 

35  B 

36  B  A 

37  A  E 

38  B  A 

39  A  D 

40  B  D 

41  A  B 

42  A  B 
4^  B  A 


D  C  D 

A  A  B 

BCD 
B  B  B 

BAB 
D  C 


DBA 


D  C  A 

A  A  B 

D  B 

A  A 

C  D 
D  C 

B  B  B 

D  B 

D  B  B 

B 

A 

D  B 

D  C  B 

D  D  B 

D  C  B 

CAB 
B 


B  B  B 

D  C 
B 

D 

A  C  B 

D  C 


B  A 

B  A  C 

AAA 
A  C  B 

A  D  B 

ABA 
BAD 
A  A  D 

A  B  D 

A  A 

A  A 

B  B  D 

B  A 

B  A 

B  A 

A  A  C 

AAA 
ADA 
A  D  7 

B  A 

BAD 
B  A 

ABB 
ADA 
A  B  D 

A  C  B 

BAD 
ADA 
A  A  D 

BCD 
B  D 

B 

BAB 

B 

A  B 

ADA 
BAD 
ADA 
B  D 

A  A  D 

AAA 
BAD 


B 

B  C  B 

ABA 
C  D  A 

C  B  A 

ADA 
C  B 

C  A 

C  A 

A 
A 

BAB 

B 

B 

A 

C  A  A 

ABB 
ABA 
A  A 

C  A  A 

C  B 

B 

B  B  B 

ABA 
CAB 
A 

C  A 

ABA 
C  B  A 

BAA 
C  B  A 

A 
B 
A 
A 

B  C  A 

C  B 

ADA 
B 
A 

C  B  A 

C  B 


B 

ADC 

AAA 
C  D  C 

B  B  B 

D  A  A 

D  D  C 

D  A  A 

C  A  A 

A 

D  A  A 

B  D  C 

A 
B 

B  D  C 

ACC 
B 

E  D  C 

E  B 

D  D  C 

ADC 
A 

ABB 
E  D  C 

B  D  D 

D  B 

D  D 

E  D  C 

A  A  C 

C  .  B  B 

C  D  C 

BAA 
A  B 

A  D 

C 

E  A  B 

ADC 
D  B 

D 

B  D 

B  A  C 

ADC 


b  D 
A  A  B 

BAB 
D  A  C 

BAB 
B  B  B 

B  D 
A  B 

B  B  A 

A  A 
ABA 
C  B  B 

B  D 
B  C 
ABC 
AAA 
B  B 
B  A  E 

A  E 
B  A  C 

B  B  A 

B  A 
B  B  A 

B  A  E 

A  A  C 

A  C 
A  C 

B  A  E 

BAA 
C  B  C 

BAB 
BAB 
B  C 
A  A 
B  A 
CAE 
B  A 
A  C 
B  C 
A  C 
BAD 
B  A 
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Response  Question 

No.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17 


44  B  B 

45  A  C 

46  B  A 

47  B  D 

48  B  B 

49  A  C 

50  A  A 

51  B  A 

52  A  B 

53  A  C 

54  B  A 

55  B  C 

56  A  B 

57  A  B 

58  A  B 

59  A  C 

60  A  A 

61  B  A 

62  B 

63  A  A 

64  A  C 

65  B  B 

66  B  D 

67  A  B 

68  B  B 

69  B  E 

70  A 

71  B  B 

72  A  C 

73  A  E 

74  A  A 

75  A  A 

76  B  A 

77  A  B 

78  A  B 

79  A  E 

80  B  A 

81  B  B 

82  A  C 

83  B  C 

84  B  A 

85  B  B 

86  B  B 

87  B  C 

88  B  B 

89  B  A 

90  A  C 

91  A  A 


B 

D 

D 

B 

BAB 
D  C  A 

D  C 

D  B  B 

B  B  B 

D  C 

B 
D 


B 

D  C  A 
C  B  A 


D 

A  A  B 
D  C  C 


B 

B 

B 


D  C  C 
A  A  B 


D 

ABA 

C 

D 


A  D  B 

A  C 

A  A 

B  D  A 

A  C  A 

B  B 

A  A  D 

A  A  B 

AAA 
ABA 
A  B  D 

A  A  D 

BAD 
A  C  B 

A  A  D 

A  A  D 

A  A  D 

A  A 

ABB 
ABA 
A  D  B 

BAD 
B  D 

ABC 
A  A 

A  A  B 

A  B 

A  A  B 

A  C  D 

A  A  D 

A  C  A 

A  A 

B  A 

A  A 

A  A 

ABB 
B  A 

A  A 

A  A  D 

A  A 

A  E  A 

A  C  D 

B  B 

A  A  B 

A  E  A 

A  C  D 

B  A 

ABC 


A  D  B 

A 

A  B 

B  B  B 

A 
A 
A 

AAA 
ABA 
C  A  A 

C  B 

CAD 
B  B  A 

C  A 

A 
A 
A 

B  B  B 

AAA 
A 

C  B 

B 

B  D  A 

ABB 
B  B  A 

A 
A 

C  B  A 

C  A  A 

A  C  B 

A 
A 
A 
A 
A 
B 
A 

CAB 

A 

ABA 
C  A 

B 
A 

B  B  B 
B  A 

A 

BAA 


C  B 

E  A  A 

C  A  C 

C  B 

A  D 

C  B 

BAA 
E  B  A 

B  D  C 

ADC 
A  A  B 

C  B  B 

C  D  C 

B  D 

C  D 

D  B 

E  A  A 

D  B 

B  D  D 

D 

D  C  B 

C 

BAB 
A  D 

A  B 

C  B  C 

D  D  C 

C 
A 

D  C 
C  B 

C  B 

E  C 

A 

C  B 

ADC 
C  A  C 

B  D  A 

C  D  C 

C 

B  B 

BAB 
E  C 

D  A  A 

E  A  A 


A  D 
A  C 
B  B  A 

B  E 
BBC 
A  D 
A  C 
A  C 
D  B  C 

B  A  E 
A  A  B 

A  C 
DAB 
BAD 
B  A 
A  B 
A  C 
A  E 
B 

B  A  E 

A  D 
B  A 
B  D 
A  A  E 

B  C 
BAB 
A  A 
B  A 
B  A  C 

CAE 
B  C 
A  C 
ABC 
B  A 
B  A 

B  A 
A  C 
ABC 
BAA 
B  A  C 

A  D 
B  D 
B  C 
A  B  D 

A  D 
BAA 
A  A  E 
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Response  Question 

No.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17 


92  A  D 

93  A  B 

94  A  B 

95  A  C 

96  B  D 

97  A  D 

98  A  A 

99  A  B 

100  B  A 

101  A  D 

102  A  D 

103  A  E 

104  A  B 

105  A  B 

106  A  B 

107  B  A 

108  A  B 

109  A  C 

110  A  D 

111  A  A 

112  B  B 

113  B  A 

114  B  A 

115  B  B 

116  A  B 

117  A  D 

118  A  D 

119  A  C 

120  A  C 

121  B  B 

122  B 

123  B  A 

124  A  A 

125  A  C 

126  B  B 

127  A  B 

128  B  D 

129  A  C 

130  B  B 


DAB 
A  B 

D  C 

A  A  B 

D  B 

B 

D  C  A 

B  B  B 

D  C 

BBC 

D 

D  C  B 

D 

A  A  D 

D 

B 

D  C  B 

D  C  B 

A  C 


D  C  A 
ABA 
A  A  B 
A  A  B 
B  C  B 
D 

D  C  B 


A  C  B 

C  B  B 

A  A  B 

D 

A  A  B 


ADD 
AAA 
A  A  D 

ABA 
ADA 
AAA 
ABA 
A  A  B 

BAD 
A  B 
A  D  B 

A  B  D 

A  A  D 

A  B  D 

ABA 
BAD 
A  A  B 

A  C  D 

B 

A  C  A 

A  B  D 

A  B  D 

A  C  D 

BAD 
AAA 
AAA 
AAA 
A  B  D 

A  B  D 

B  B  D 

ABA 
A  C  B 

A  A 

B  A 

B 

ABA 
A  C  D 

ADD 
A  A  D 


A  B 

B  A 

C  A 

ABA 
ABA 
B  B  A 

C  B  A 

BAA 
C  B 

A 

C  A  A 

A 

C  B  A 

A 

ADA 

B 

A 

C  A  A 

A 

C  D  A 

C  A  A 

A 
A 

C  A  A 

AAA 
ABA 
ABA 
CCA 
A 

C  B  A 

BAA 
ABA 
A 

A  A 

A 

A  C  A 

B  C  A 

CCA 
C  A  A 


E  D  C 

DAB 
C  D  C 

C  A  A 

E  D  C 

D  B 

E  D  B 

BAB 
C  D  C 

E 

D  D  C 

D  D 

C  D  C 

D  D 

BAA 
B  D 

C  B 

C  D  C 

C  D  C 

E  A  C 

E  A  A 

E  D 

E  D 

D  A  A 

CAB 
D  A  A 

D  A  A 

E  D  C 

E  D 

C  D  C 

BAB 
D  B 

BAA 
C  A  A 

C  B  B 

DAB 
D  D  B 

D  D  D 

E  D  C 


A  D 
A  D 
B 

B  A  E 

B  A  E 

A  D 
B  A  E 

BAB 
B  B 
A  E 
C  A  C 

A  D 
A  A  C 

A  C 
DAB 
B  C 
B  B 
B  A  E 

B  A  C 

B  B  B 

A  A  D 

A  D 
A  E 
BBC 
B  B  B 

B  B  D 

B  B  D 

B  B  A 

A  E 
B  B  B 

BAB 
B  A  C 

B  B  A 

B  B  B 

B  A  C 

A  A  D 

CAD 
B  A  C 

BAD 
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Response  Question 

No.  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34 


1 

2  C  C 

3 

4  A  A 

5  B  B 

6  B  B 

7  C  A 

8  D  C 

9 

10  D  C 

11 
12 

13  D  C 

14 

15 

16 

17  C  C 

18 

19  D  B 

20  B 

21  D  C 

22  D  C 

23 

24  B  B 

25  D  B 

26  A  B 

27  B 

28  D  C 

29  D  B 

30  A  C 

31  B  B 

32  D  C 

33  A  A 

34  B 

35  D 

36 

37  A  B 

38  D  C 

39  B 

40 

41  D 

42  A  C 

43  D  C 


B  C 
CAB 

BAG 
DAB 
BAD 
ABC 
B  D 
B 

B  A 
A  A 
B  A 
C  B  B 

B  D 
B  C 
B  C 
AAA 
A  B 
B  A  E 

A  E 
A  A  C 

B  A 
B  C 
B  B  A 

B  A  E 

BAG 
B  B 
ABC 
B  A  E 

B  B  A 

C  B  A 

BAB 
B  B 

B  C 
A  D 
B  A 
BAD 
B  A 
A  E 
B  D 
A  A 
A  A  D 

A  B 


D  C  A 

A  A  B 

B  C  B 

B  B  B 

C  A  A 

D  C 

D  C 


D  C  C 


CCA 
A  A  B 

D  B  B 

A  A 

D  C  B 

D  C 

B  B  B 

D  B  B 

A  C  B 

B 

D  C  A 

D  B  B 

D  C  B 

D  C  B 

D  C  B 

B 

D 

B  B  B 

D  C 

B 

D 

A  C  B 

D  C  B 


B  C 

A  C  D 

A  C  A 

A  C  B 

A  E  B 

ADA 
B  D  D 

B 

A  C  A 

A  A 

B  A 

B  B  D 

B  E 

B  B 

A  A  C 

A  C  A 

A  E  D 

A  E  A 

A  B  D 

ADD 
A  B 

BAB 
A  E  D 

A  C  A 

A  D  B 

D  D 
A  E  D 

A  A  D 

BAD 
A  C  D 

B 

B  C  B 

A  B  D 

A  C 

ADA 
BAD 
A  E  B 

B  D 

ADD 
ADA 
BAD 


A 

B  B  A 

ABA 
C  B  A 

B  B  A 

ABA 
C  B 

A 

ABA 

A 

A 

C  C  B 

B 
B 
A 

C  A 

ABA 
C  B  A 

A  A 

C  B  A 

C  A 

A 

B  B  B 

C  B  A 

C  B  A 

A 

C  A  A 

C  B  A 

C  B  A 

C  B  B 

C  B  A 

A 
B 
A 
A 

B  B  A 

C  B 

A 
B 
A 

C  B  A 

C  B 


E 

ADC 

D  A  A 

C  B  B 

EBB 
D  A  A 

D  D  C 

C  A  A 

D  A  A 

A 

C  B 

B  D  C 

B 
E 

C  D  C 

ACC 
E  A  A 

E  B 

D  li 

B  D  C 

E  D  C 

A 

ABB 
E  D  B 

E  A  C 

E  B 

E  D  C 

E  D  B 

BAG 
ADC 
B  D  C 

C  A  A 

B  6 

D  D 

E 

DAB 
ADC 
E  B 

D 

E  D 

E  A  C 

ADC 
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Response  Question 

No.  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34 


44  B 

45 

46 

47  A 

48  B 

49  B 

50  D 

51  B 

52 

53  B 

54  D 

55  D 

56  B 

57  B 

58  D 

59  D 

60  D 

61  B 

62 

63  A 

64  B 

65  D 

66 

67  C 

68 

69  A 

70  D 

71 

72  D 

73  D 

74 

75  B 

76 

77 

78 

79 

80 

81  B 

82  D 

83 

84  D 

85  D 

86 

87  D 

88  D 

89  C 

90  C 

91  A 


A 


A  B 
C  A 
C  B 
D  B 
B  B 
C 


A  B 
C 

B  C 
B  B 


C  B 
C  B 


C  A 

A  B 

C 


B  A 


A  A 


A  E 
A  C 
B  A 
B  E 
B  D 
A  D 
A  B 
A  C 
B  C 
A  E 
A  C 
B  C 
A  B 
A  C 
A  C 
A  D 
A  C 
A  E 
B 

A  D 
A  B 
B  A 
B  D 
A  C 
B  C 
B  A 
A  E 
A  C 
A  C 
A  D 
B  C 
A  C 
B  C 
A  D 
A  D 
A  E 
B  A 
A  C 
A  D 
B  C 
A  B 
A  D 
B  D 
B  C 
A  D 
A  D 
A  B 
A  C 


B 


A  A 

B 

B 

D 

B 

B  A 
D  C 
D  C 
A  A 
A  A 
D  C 
B 
D 
B 

A  A 
B 

D  C 

C  B 

B 

D 

A  A 
A  A 
D  C 

B 


C 

B 

D  C 

D  A 
D  C 

D 

A  B 
D  B 
C 

A  A 


A  E  C 

A  C 

A  C  B  B 

B  E  A  A 

B  D  A  C  B 

A  C  B 

A  A  D 

A  C  B 

B  D  A  A  B 

B  A  E  B  A  B 

A  A  C  D  C  C 

A  D  D  C  B 

B  B  A  B  B 

B  A  D  A  A  B 

B  A  D  C 

A  C  B 

ADD 
A  E  B 

B 

B  B  B 

A  A  B 

B  A  D  C 

B  D 

C  A  E  C  B  C 

B  B 

B  A 

A  C  D 

A  C  B 

B  A  C  A  A  B 

B  A  C  D  C  C 

B  C 

A  D  B 

A  D  C  A 

A  D  B 

A  D  B 


B 

B 


A 


A 


B  A 

A  D  B 

A  D  D  C  C 

A  C  A  C  B 

A  A  D  A  B 

A  C  D  C 

B  C 

A  D  A  A  B 

A  D  A  B  A 

AEG 
A  D  A  A  B 

A  D  A  A  A 


A 

A 

A 

B 

B 

A 

A 

A 

B 

A 

B 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

B 

B 

A 

A 

A 

A 

A 

A 

A 

B 

A 

A 

A 

A 

A 

B 

A 

B 

B 

A 

A 

A 

A 

A 

A 

A 

A 


E  B 

E 

C  B 

E  A  A 

D  B 

D  B 

C  D 

D  A  A 

C 

E  B  A 

EDO 
D  D  C 

E  A  A 

E  A  A 

D  D  C 

D  B 

E  D 

E  B 

B  B  B 

E  A  A 

D  B 

ADC 
D 

D  C  B 

A  A 
DAB 
E  B 

E  B 

C  A  A 

E  D  C 

D 

E  B 

C  C 
E  B 

E  B 

E  C 

A 

E  B 

ADC 

C 

C  A  A 

D  D  C 

D  B  B 

E  A  A 

DAB 
E  B 

D  B 

D  A  A 
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Response  Question 

No.  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34 


92  D  C 

93  A  B 

94  D  C 

95  A  A 

96  D  B 

97  B 

98  A  A 

99  A  B 

100  D  D 

101 

102  D  C 

103  D 

104  D  C 

105  D 

106  A  A 

107  D 

108 

109  D  C 

110  D  C 

111  A  C 

112  A  A 

113  D 

114  D 

115  D  C 

116 

117 

118 

119 

120  D 

121  D  C 

122  A  A 

123  B 

124 

125  A  A 

126  B  B 

127  A  B 

128  D  C 

129  A  C 

130  D  C 


A  B 
A  C 
B 

BAD 
B  A  E 
B  D 
C  D 

BAG 
B  B 
A  E 
A  A  C 

A  E 
B  A  E 

A  C 
DAB 
B  C 
B  B 
A  A  C 

B  B 

B 

DAD 
B  B 
B  B 
ABC 
B  B 
B  D 
B  D 
B  A 
A  E 
B  B  B 

BAB 
B  B  A 

B  A 
B  B 
BBC 
A  B  D 

CAD 
BAG 
DAE 


D 

A  B 

D  C 

A  A  B 

D  B  B 

B 

D  C 

D  C  B 

B  D  D 

D  C  A 

D 

D  C  B 

D 

A  A  D 

D 

D  C  B 


A  A  B 


D  C  A 


D 

D  C  B 
A  A  B 


A  A 

A  C  A 
C  B  C 
A  C  B 
D  C  B 


A  E  D 

A  C  A 

BAD 
ADA 
A  E  D 

B  D  B 

ADD 
A  C  D 

BAD 
A  C 

ADD 
A  E  D 

A  C  D 

ADD 
ABA 
BAD 
B  B 

A  C  D 

B 

A  E  A 

A  E  A 

ADD 
A  E  C 

ADA 
ADA 
ADA 
ADA 
A  C  D 

ADD 
A  C  D 

B 

A  E  B 

ADA 
ADA 
A  E  B 

B  C  A 

A  E  D 

ADA 
A  C  D 


A 

B  A 

C  A 

ABA 
C  B  A 

B 

C  A  A 

C  B  A 

D  A 

A 

C  A  A 

A 

C  B  A 

A 

ADA 

B 

A 

C  B  A 

A 

C  B  A 

AAA 
A 
A 

C  A  A 

B  D  A 

ABA 
ABA 
CCA 
A 

C  B  A 

B 

B  A 
ABA 
ABA 
B  B  A 

C  A  A 

C  C  B 

ABA 
C  D  A 


E  D 

E  A  B 

E  D  C 

E  A  A 

E  D  B 

D  B 

E  A  A 

EBB 
C  D  D 

C 

E  D  C 

E  D 

E  D  C 

E  D 

BAA 
D  D 

E  B 

D  D  C 

E  D  C 

E  A  C 

E  A  A 

E  D 

E  D 

D  A  C 

C  B  B 

DAB 
E  A  B 

D  D  C 

E  D 

C  D  C 

E  B 

E  A  A 

D  A  A 

EBB 
E  A  C 

D 

C  A  A 

E  D  C 
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Response  Question 

No.  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51 


1  B 

2  D  A 

3 

4  B  A 

5  B  A 

6  B  A 

7  B  B 

8  B 

9  C  B 

10  B  A 

11  A 

12  A  B 

13  C  B 

14  B 

15  B 

16  A  B 

17  A  A 

18  B  B 

19  B  A 

20  A 

21  B  A 

22  B  A 

23  A 

24  B  B 

25  B  A 

26  B  A 

27  A 

28  A  A 

29  B  A 

30  B  A 

31  B  B 

32  B  A 

33  B  A 

34  B 

35  A 

36  A 

37  B  A 

38  B 

39  A 

40  B 

41  A 

42  A  A 

43  B 


B 

B  D  A 

E  A  A 
C  B  B 

D  B  B 

C  D  C 

D  D  C 

GDC 

A 

A 

B  D  C 

B 
E 
B 

ACC 

A 

E  D  B 

D  B 

D  D  C 

C  D  C 

A 

ABB 
E  D  B 

D  A  C 

B  B 

E  D  C 

E  D  B 

B  A  C 

ABB 
B  D  C 

C  A  A 

A  B 

A  D 

C 

DAB 
ADC 
E  B 

D 
D 

C  A  C 

ADC 


B  B 
C  B  A 

B  A  C 

B  A  C 

BAD 
A  A  C 

B  D 
A  A 
A  C 
A  A 
A  A 
C  A  C 

B  A 
B  C 
A  B 
AAA 
A  A 
B  A  E 

A  C 
BAB 
A  C 
A  A 
B  B  A 

B  A  E 

BAA 
A  B 
A  B 

B  A  E 

BAB 
BBC 
B  A  C 

B  A  E 

B  B 
A  A 
A  C 
B  A  C 

B  A 
A  C 
B  D 
A  D 
AAA 
A  A 


D  A  C 

A  A  B 

BBC 
B  B  B 

D  C  A 

D  C 

D  C 

A  A  B 

B 

C  B  C 


D  C  A 

CCA 

D  B  B 

B 

C  A 
D  C 

B  B  B 

D  B  B 

A  C  D 

B 

D  C  A 

D  B  B 

A  C  B 

B  D  A 

D  C  B 

A  A  B 

B 
D 

ABB 
D  C 

B 

D 

A  C  A 

D  C 


B  C 
A  A  B 

A  C  A 

A  D  B 

A  C  B 

A  C  D 

B  D  D 

A  A  D 
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Appendix  C:  Survey  Conanents 


USAF  Survey  Control  Number  SCN  91-12 


This  appendix  is  a  complete  list  of  all  comments 
received  (in  raw  form)  with  the  survey  responses. 
Additional  information  was  added  in  square  brackets. 

Other  Documentation  Needed 

Use  AFOTEC  Pamphlet  800-2  vol .  3  as  a  guide  during 
devel opment . 

The  other  documents  required  for  software  maintenance  are 
software  problem  reports  or  software  deficiency  reports. 

The  software  and  system  engineering  design  rationale  (why 
it  was  done)  is  often  required  to  prevent  redundant 
development  efforts  later  in  the  life  cycle  and  to  assist 
in  understanding  the  design.  This  information  can  be 
captured  in  software  development  files. 

ECP’s  and  SCNs  are  used  by  i the  contractors  in  OFP 
development,  but  not  by  the  ALC  for  code  maintenance. 

Non-Complex  Computer  Specifications  that  are  updated  for 
current  configuration. 

Flow  charts,  Detailed  Test  Instructions,  Test  procedure 
instructions . 

D I -QCIC-8057 2  [Software  Quality  Program  Plans. 

MATE  STDs 

DI -MCCR-80011  [Software  Standards  and  Procedures.  This 
document  was  superseded  by  the  Software 
Development  Plan  (SDP),  DI-MCCR-80030A , 
which  was  covered  by  the  surveys. 

Technical  orders  (T.O.s) 

Vendor  Hardware  Manuals  for  COTS  hardware/software. 

For  ATE  (Automatic  Test  Equipment)  Maintenance:  Source 
code  on  deliverable  media.  Schematic  of  unit  under  test 
(UUT)  and  interface  devices  (ID),  top  assembly  drawing  and 
parts  list  of  UUT  and  ITA,  and  engineering  data/spec 
drawings  for  unique  devices  (i.e.  PAL,  PROM  data). 

Test  Requirements  Document. 
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Computer  Program  Identification  Number. 

Technical  Orders. 

Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP). 

Other  documents  required  are  a  Software  Development  Manual 
(How  do  you  change,  modify,  fix  this  software's  source 
code,  then  regenerate  the  executable?)  and  Special 
Development  Considerations  (What  special  knowledge  does  a 
programmer  need  to  develop  this  software).  Many  of  my 
software  documents  are  very  sketchy.  They  don't  begin  to 
provide  the  information  needed  to  understand  the  design  and 
operation  of  a  CPCI .  Furthermore,  if  the  software 
interfaces  to  hardware,  the  interface  is  not  clearly 
explained . 

High  Order  Language  Standards,  Test  Requirements  Documents 
(TRDs),  and  Engineering  Support  Data  ( DI -MCCR-80 285 ) . 

The  Software  Integration  Requirements  Document  (SIRD)  is  a 
document  that  the  jcontractori  uses.  This  is  the  most 
important  document  in  the  development  process  and  is  not 
deliverable  to  the  government. 

The  SDFs  which  have  not  been  a  CDRL  or  DID  item  contai  i  the 
rationale  behind  the  design  of  the  software,  which  is 
critical  for  effective  software  support  on  complex  or  large 
systems . 


General  Comments 

By  focusing  on  "maintenance  only"  and  defining  it  narrowly, 
the  results  of  this  survey  may  be  difficult  to  interpret. 
The  questions  do  not  recognize  the  documentation 
requirements  for  maintenance  related  activities  such  as 
establishing  an  organic  capability;  monitoring  contractor 
development  activities;  fielding  the  software;  and 
operating  the  system.  Therefore,  answers  provided  for 
documents  such  as  the  CRISD,  STP  and  SCN  may  be  misleading 
if  respondents  do  not  expand  their  focus. 

I  answered  based  on  personal  experience  at  an  ALC  as  well 
as  personal  experience  (of  almost  20  yrs)  listening  to  ALC 
software  maintainers  talk  about  what  they  need  to  do  their 
job.  I  have  not  personally  worked  on  software  maintenance 
projects  (other  than  pc  type)  for  several  years. 

I've  never  heard  of  most  of  these  documents.  I  don't  know 
if  any  of  these  documents  were  tailored  to  meet  maintenance 
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requirements,  and  if  they  were,  who  participated. 

Therefore,  I  could  not  answer  question  #5  for  most  cases. 
The  ones  I  did  answer  were  based  only  on  my  knowledge  of 
the  document  and  not  any  tailoring. 

As  a  S/W  and  F/W  development  and  maintenance  group,  we 
typically  have  to  work  with  S/W  programs  and  systems  that 
are  poorly  documented  or  sometimes  not  documented  at  all. 

I  attribute  this  to  the  acquisition  cycle  and  the 
acquisition  office  "Saving  Money"  by  not  buying  the 
documentation  on  the  front  end  and  often  having  "Contractor 
Maintenance"  sustain  the  S/W  for  several  years  after 
acquisition.  This  causes  two  long  term  problems;  1.  Loss 
of  configuration  control  of  the  S/W  and  2.  Lack  of 
meaningful  documentation  for  the  S/W  system  (if  the 
documentation  was  ever  acquired).  With  these  two  problems 
under  consideration  the  long  term  maintenance  of  the  S/W 
system  becomes  extremely  costly  due  to  the  need  to  reverse 
engineer  the  deliverables  in  order  to  enhance,  change  or 
sustain  the  S/W.  The  original  acquisition  office  "Doesn't 
Care"  because  the  "Did  their  job"  in  acquiring  the  system 
(at  a  good  price)  and  now  are  working  on  "New  Systems". 
Meanwhile  "another"  section  or  branch  of  the  acquisition 
office  has  to  foot  the  bill  for  maintenance  and  organic 
support.  Force  acquisition  offices  to  use  the  DOD 
standard ! ! ! 

Updates  to  the  type  of  software  that  we  maintain  should  be 
supportable  if  design  documents,  user  manual  and  test 
documentation  are  available.  Configuration  control  is 
through  CPIN  system,  not  Version  Description  Document. 
DOD-STD-2167A  and  documentation  also  allows  government 
control  and  insight  into  contractual  effort. 

The  software  type  and  environment  plays  an  important  role 
in  documentation  requirements  for  maintenance.  Believe 
should  be  differentiation  between  weapon  system  software 
and  records/payrol 1 /accounting . 

My  organization  is  primarily  performing  hardware  and 
software  development  for  a  space  system.  All  our  work  to 
date  has  involved  reverse  engineering  the  system  with  very 
little,  to  no,  documentation. 

It  seems  to  me  that  many  of  the  documents  that  we  require 
are  unnecessary  and  if  properly  thought  out,  we  could 
reduce  the  amounts  required  of  contractors  and  require  more 
detail  in  the  documents  we  receive.  This  would  be  even 
better  if  the  means  documents  are  delivered  was  electronic 
rather  than  hardcopy. 
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Survey  is  really  applicable  to  system  operational  software 
and  not  ATE  test  programs.  Items  answered  on  scan  sheet 
were  educated  guesses  as  to  operational  software  needs. 

The  iprojecti.  involved  reverse  engineering  the  controller 
using  a  HEX  microcode  listing,  a  set  of  electronic 
schematics,  the  technical  order,  and  a  partly  commented 
program  listing  in  basic.  Our  emphasis  in  this  particular 
effort  was  not  involved  with  2167A.  Our  participation  in 
this  survey  is  therefore  marginally  valid. 

Question  92  has  both  answers  marked.  My  organization  is  an 
ALC  organization.  However,  we  are  both  the  SPO 
(acquisition  agency)  and  maintenance  organization  for  many 
ground  electronic  warfare  weapon  systems. 

TRDs  for  digital  TPS  development  have  often  been  deemed 
useless.  Analog  TRDs  have  been  useful.  Let's  tell  the  SPO 
this  and  use  this  saved  money  to  buy  COTS  ATE  which  is  by 
far  the  best  TPS  development  tool.  SPOs  should  "pool" 
their  resources  together  on  like  systems  instead  of  telling 
the  depot  they  don't  have  dollars  for  a  tester. 
Individually,  that  probably  is  true  but  together  great 
things  can  be  accomplished. 

All  documents  that  were  marked  as  "(b)  not  available"  will 
be  available  later  in  our  program. 

Information  on  tailoring  available,  but  in  a  different 
location.  Consequently  marked  "don't  know."  Similar 
reasoning  for  some  documents  marked  "not  available." 
ECPs/SCNs  were  accomplished  for  this  project,  but  not 
applicable  for  software  I  am  cognizant  of. 

The  particular  software  we  maintain  is  very  old,  often 
predating  the  use  of  such  documents.  There  are  internal 
counterparts  to  many  of  the  documents. 

Answers  to  this  survey  were  derived  from  experience  on 
several  projects.  My  involvement  on  any  one  project  did 
not  include  all  aspects  represented  by  the  documents 
listed.  Further,  not  all  documents  were  available  for  any 
one  project . 

We  applied  heavy  tailoring  to  the  SRS  and  SDD  to 
accommodate  object  oriented  design  for  Ada.  We  used  CASE 
tools  to  prepare  these  documents.  2167A  is  not  easy  to  use 
in  these  circumstances.  It  asks  for  too  much  information 
that  IS  automatically  generated  and  checked  by  the  CASE 
tool.  Ada  specs  tell  all  we  need  about  internal 
interfaces.  System  diagrams  and  external  interfaces  are 
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much  more  important.  We  use  equivalent  documents  for  the 
following  IRS/IDD,  STP/STD/STR,  CSOM,  SUM,  FSM  (commercial 
off  the  shelf),  ECP/SCN. 

The  documentation  used  here  was  developed  with  many  DIDs 
rolled  together  into  one  document.  This  improves 
traceability  and  "defragments"  the  document;  however,  the 
content  of  the  DIDs  is  distorted  and  in  many  cases  lost. 

The  Software  Development  Plan  and  the  Software  Test  Plan 
are  not  needed  for  maintenance.  They  are  vital  documents 
from  a  contract  monitoring  perspective.  The  provide  an 
insight  into  whether  the  contractor  really  understands  the 
task  at  hand  and  provide  a  means  for  the  government  to 
correct  deficiencies  early  in  the  contract. 

One  thing  that  seems  to  be  missing  in  all  documentation  is 
an  explanation  of  algorithms  used.  Many  times  we  must 
reverse  engineer  the  software  to  understand  what  the 
contractor  was  trying  to  do  when  complex  equations  are 
used . 

We  have  developed  our  own  in-house  TPS  maintenance  guide. 

My  organization  performs  Automatic  Test  Equipment  software 
maintenance.  Obviously,  this  is  not  what  2167  was  written 
to  support.  Current  development  practice  is  to  follow 
2167,  tailoring  where  necessary.  This  practice  doesn't 
address  most  of  my  maintenance  because  the  initial 
development  happened  over  the  past  20  years  and  2167  didn't 
exist  or  wasn't  used.  Additionally  there  are  several 
alternative  and  in  some  cases  general  procedures  we  follow. 

Because  the  iprograiul  is  not  in  FSD  at  this  time,  these 
documents  are  not  available.  Maintenance  content  of  these 
deliverables  is  not  known  at  this  time  since  the  iprogramc 
has  not  entered  FSD. 

I  can't  fill  out  your  survey  because  your  DID  numbers  are 
new  and  haven't  filtered  down  to  the  working  level  yet.  I 
can't  answer  maintenance  questions,  but  can  tell  you  that 
ALCs  almost  never  have  input  into  DID  selection. 


80 


Questions  from  AFOTECP  800-2  Vol  3,  31  October  1989 

D-1.  A  useful  software  documentation  master  list  is 
available . 

D-2.  Each  physically  separate  part  of  the  documentation 
includes  a  useful  table  of  contents. 

D-3.  Each  physically  separate  part  of  the  documentation 
includes  a  useful  index. 

D-4.  Each  physically  separate  part  of  the  documentation 
includes  a  useful  list  of  major  terms  and  acronyms  used  in 
that  document . 

D-5.  The  documentation  has  been  physically  separated  into 
(sets  of)  volumes  each  with  a  distinct  part. 

D-6.  Ma]or  parts  of  the  documentation  are  essentially 
sel f -contained . 

D-7.  A  numbering  scheme  has  been  adopted  which  allows  for 
easy  addition  or  deletion  of  narrative  parts  of  the 
documentation  and  graphic  materials. 

D-8.  Graphic  materials  (figures,  charts,  lists,  etc..)  are 
physically  separate  (e.g.,  on  separate  pages)  from 
narrative  description. 

D-9.  The  documentation  includes  a  separate  part  for  the 
description  of  external  interfaces. 

D-10.  The  documentation  adequately  describes  the  external 
interfaces  . 

D-11.  The  documentation  includes  a  useful  version 
description  document. 

D-12.  The  documentation  includes  a  separate  part  for  the 
description  of  each  major  function. 

D-i3.  The  documentation  adequately  describes  each  major 
system  function. 

D-14.  The  documentation  adequately  describes  how  program 
initialization  is  performed. 
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D-15.  The  documentation  adequately  describes  how  program 
termination  is  performed. 

D-16.  Recovery  from  externally  generated  error  conditions 
is  adequately  described  in  the  documentation. 

D-17 .  Recovery  from  internally  generated  error  conditions 
is  adequately  described  in  the  documentation. 

D-18.  The  documentation  includes  a  separate  part  for  the 
description  of  the  program  data  definitions. 

D-19.  The  documentation  adequately  describes  program  data 
def ini t i ons . 

D-20.  The  global  data  master  list  includes  useful 
information  about  each  global  data  item  such  as  type, 
range,  scaling,  units,  usage,  etc. 

D-21.  Inputs/outputs  for  each  program  are  adequately 
described  in  the  documentation. 

D-22.  Program  data  partitioning  is  adequately  described  in 
the  documentation. 

D-23.  The  procedures  for  altering  basic  data  storage  sizes 
are  adequately  explained  in  the  documentation. 

D-24.  The  documentation  adequately  explains  any  use  of 
shared  memory . 

D-25.  The  timing  scheme  designed  for  the  program  is 
adequately  explained  in  the  documentation. 

D-26.  The  documentation  adequately  describes  the  timing 
requirements  for  each  major  function  of  the  program. 

0-2"^  .  .^ny  dynamic  allocation  of  resources  is  explained  in 
the  documentation. 

D-28.  The  documentation  adequately  explains  how  interrupts 
are  processed. 

D-29.  The  documentation  adequately  describes  memory 
allocation  for  the  program. 

D-30.  Storage  requirements  for  each  major  function  of  the 
program  are  adequately  described  in  the  documentation. 

D-31.  The  documentation  adequately  explains  how  external 
I/O  IS  processed. 
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D-32.  The  documentation  adequately  describes  how  the 
control  flow  is  organized. 

D-33.  The  documentation  adequately  describes  the  purpose 
of  each  module. 

D-34.  Parameters  for  each  module  are  adequately  described 
in  the  documentation. 

D-35.  The  order  of  arguments  for  this  program  as  described 
in  the  documentation  corresponds  to  the  order  of  arguments 
as  shown  in  this  programs  source  listing. 

D-36.  Data  for  each  module  are  adequately  described  in  the 
documentat i on . 

D-37.  The  processing  done  by  each  module  is  adequately 
explained  in  the  documentation. 

D-38.  Special  processing  considerations  of  each  module  are 
adequately  explained  in  the  documentation. 

D-39.  The  use  of  any  complex  mathematical  model 
(technique,  algorithm)  is  adequately  explained  in  the 
documentation . 

D-40.  Any  use  of  recursive  or  reentrant  programming  is 
adequately  described  in  the  documentation. 

D-41.  A  useful  set  of  standards  has  been  followed  for  the 
development  of  the  documentation. 

D-42.  A  set  of  useful  standards  has  been  followed  for  the 
construction  of  all  flowcharts  or  procedural  processing 
descriptions . 

D-43.  Documentation  of  each  major  functional  part  of  the 
program  follows  the  same  format. 

D-44.  The  format  of  the  documentation  reflects  the  program 
structure . 

D-45.  The  documentation  is  organized  as  a  systematic 
description  of  the  program  from  levels  of  less  detail  to 
levels  of  more  detail. 

D-46.  Each  part  (sentence,  paragraph,  subsection,  section, 
chapter,  volume,  etc.)  of  the  documentation  tends  to 
express  one  central  idea. 


83 


D-47.  The  terminology  used  in  the  documentation  to 
describe  the  program  is  easily  understood. 

D-48.  The  program  test  plan  is  adequately  described  in  the 
documentation . 

D-49.  A  useful  set  of  test  procedures  for  high  levels  of 
program  testing  is  contained  in  the  documentation. 

D-50.  A  useful  set  of  test  procedures  for  low  levels  of 
program  testing  is  contained  in  the  documentation. 

D-51.  The  limitations  and  incompleteness  of  the  test 
procedures  are  described  in  the  documentation. 

D-52.  The  sample  test  data  are  adequately  described  in  the 
documentat i on . 

D-53.  Program  support  tools  that  aid  in  testing  the 
program  are  adequately  described  in  the  documentation. 

D-54.  The  documentation  describes  software  test  probes 
that  aid  in  identifying  processing  performance. 

D-55.  System  specifications  are  easily  traceable  to  the 
actual  functions  that  implement  the  specification. 

D-56.  A  functions  description  is  easily  traceable  to  the 
detailed  descriptions  of  the  module(s)  performing  that 
function . 

D-57.  An  algorithm  described  in  the  documentation  can  be 
easily  traced  to  its  representation  in  the  source  listings. 

D-58.  Variables  and  constants  used  in  an  algorithm  can  be 
easily  traced  to  their  source  code  equivalent. 

D-59.  It  is  easy  to  trace  the  program  control  flow  at  all 
system  levels,  (program  calling  structure) 

D-60.  It  is  easy  to  trace  the  data  flow  of  the  program  at 
all  system  1 evel s . 

D'61.  Global  data  items  and  data  structures  aL*?  easily 
traceable  to  the  modules  which  use  them. 

D-b2.  Data  items  and  data  structures  in  the  database  are 
easily  traceable  to  the  modules  which  use  them. 

D-63.  Specific  information  is  easily  traceable  from 
document  to  document  and  from  document  to  source  listings. 
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D-64.  It  is  easy  to  locate  specific  information  within  the 
documentation . 


D-65.  The  documentation  organization  contributes  to  the 
maintainability  of  the  program. 

D-66.  Software  descriptiveness  as  reflected  in  the 
documentation  contributes  to  the  maintainability  of  the 
program . 


D-67.  Software  traceability  as  reflected  in  the 
documentation  contributes  to  the  maintainability  of  the 
program. 


D-68.  Overall  it  appears 
program  contribute  to  the 


that  the  characteristics  of  the 
maintainability  of  the  program. 
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Appendix  E: 


Document _T a i 1 orinq  Responses 


System/ Segment  Design  Document 


Document  was  tailored:  14 
Tailoring  not  ne‘‘‘ded:  15 
Should  have  been  tailored:  3 


Amount  of  j 

1  ALC  Ta] 

1  Yes 

L 1  or ing 

No 

- 1 

Maintenance  ] 
Information 

1 

I  too  1 1 tt 1 e 1 

1  0 

3 

1 - 1 

acceptabl e  j 

13 

5 

Other  j 

Info  rma  1 1 on 

I  too  much 

1  j 

1 

acceptabl e 

12  1 

7 

Software  Development  Plan 


Document  was  tailored:  13 
Tailoring  not  needed:  ? 
Should  have  been  tailored:  1 


Amount  of 

ALC  Ta: 
Yes  j 

L 1  or ing 

No 

Maintenance 

1 

Information  1 

I 

1 

1 

Itoo  little 

3 

6 

acceptabl e 

15 

1 

C  t-  r 

Inf  ormat 1  on 

too  much 

5 

acceptabl e 

13 

5 

1 _ 
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Software  Requirements  Specification 


Document  was  tailored:  23 
Tailoring  not  needed:  16 
Should  have  been  tailored:  3 


1 

Amount  of  j 

ALC  Ta] 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

4 

4 

acceptable 

13 

6 

Other 

Information 

too  much 

1 

4 

accept abl e 

1 

16 

1 

6 

Interface  Requirements  Specification 


Document  was  tailored:  15 
Tailoring  not  needed:  15 
Should  have  been  tailored:  4 


Amount  of 

ALC  Tad 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

3 

2 

acceptable 

7 

9 

Other 

Information 

too  much 

3 

2 

acceptabl e 

7 

9 
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Interface  Design  Document 


Document  was  tailored:  14 
Tailoring  not  needed:  13 
Should  have  been  tailored:  5 


Amount  of 

ALC  Ta] 
Yes 

L 1  or ing 

No 

Maintenance 

Information 

too  little 

2 

1 

acceptabl e 

9 

6 

Other 

Information 

too  much 

1 

1 

acceptabl e 

10 

6 

Software  Design  Document 


Document  was  tailored:  23 
Tailoring  not  needed:  16 
Should  have  been  tailored:  3 


Amount  of 

ALC  Tal 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

2 

0 

acceptabl e ' 

15 

4 

Other 

Information 

too  much 

1 

1 

acceptabl e 

16 

3 
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27 

21 

3 


Software  Product  Specification 


Document  was  tailored: 
Tailoring  not  needed: 
Should  have  been  tailored: 


Amount  of 

ALC  Ta; 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 
- 1 

4 

0 

acceptabl e 

15 

11 

Other 

Information 

too  much 

2 

1 

acceptable 

1  17 

1  10 

Version  Description  Document 


Document  was  tailored:  14 
Tailoring  not  needed:  13 
Should  have  been  tailored:  4 


Amount  of 

-  -  -  _ 1 

ALC  Ta; 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

1 

0 

acceptabl e 

8 

11 

Other 

Information 

too  much 

1 

1  0 

acceptabl e 

8 

11 
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Software  Test  Plan 


Document  was  tailored:  24 
Tailoring  not  needed:  14 
Should  have  been  tailored:  3 


Amount  of  ! 

_ 1 

ALC  Ta] 
Yes 

.  1 oring 

No 

Maintenance 

Information 

too  little 

5 

2 

acceptabl e 

11 

8 

Other 

Information 

too  much 

4 

2 

j 

acceptable 

12 

8 

Software  Test  Description 


Document  was  tailored:  26 
Tailoring  not  needed:  10 
Should  have  been  tailored:  3 


Amount  of 

ALC  Taj 
Yes  1 

L 1 oring 

No 

Maintenance 

Information 

too  little 

6  1 

2 

acceptabl e 

11  ^ 

9 

Other 

Information 

1 

too  much 

3 

2 

acceptabl e 

14 

_ 1 

9 
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Software  Test  Report 


Document  was  tailored:  19 
Tailoring  not  needed:  13 
Should  have  been  tailored:  4 


Amount  of  j 

ALC  Tai 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

3 

3 

acceptable 

11 

7 

Other 

Information 

too  much 

2 

3 

acceptable 

12 

7 

Computer  System  Operator's  Hanual 


Document  was  tailored:  21 
Tailoring  not  needed:  12 
Should  have  been  tailored:  6 


Amount  of 

ALC  Ta: 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

6 

4 

acceptabl e 

11 

8 

Other 

Information 

_ 

too  much 

2 

4 

acceptabl e 

15 

8 
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Software  User's  Manual 


Document  was  tailored:  25 
Tailoring  not  needed:  10 
Should  have  been  tailored:  7 


Amount  of 

ALC  Tai 
Yes 

iloring 

No 

Maintenance 

Information 

too  little 

3 

4 

acceptabl e 

14 

Q 

Other 

Information 

too  much 

3 

3 

accept abl e 

14 

10 

Software  Programmer’s  Manual 


Document  was  tailored:  18 
Tailoring  not  needed:  17 
Should  have  been  tailored:  5 


Amount  of  j 

ALC  Tad 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  1 i tt 1 e  j 

1 

4 

acceptabl e 

11 

6 

Other 

Information 

too  much 

2 

3 

acceptabl e 

10 

7 
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Firmware  Support  Manual 


Document  was  tailored:  16 
Tailoring  not  needed:  14 
Should  have  been  tailored:  4 


Amount  of  j 

ALC  Tai 
Yes 

j 

L 1 oring 

No 

^  1 

Maintenance 

Information 

too  little 

1  ! 

4 

acceptabl e 

9 

3 

Other 

Information 

too  much 

3 

1 

acceptable 

7 

6 

Computer  Resources  Integrated  Support  Document 


Document  was  tailored:  20 
Tailoring  not  needed:  11 
Should  have  been  tailored:  5 


Amount  of 

ALC  Ta: 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

2 

3 

acceptabl e 

12 

3 

Other 

Information 

too  much 

3 

1 

acceptabl e 

11 

5 
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Engineering  Change  Proposal 


Document  was  tailored:  18 
Tailoring  not  needed:  14 
Should  have  been  tailored:  5 


Amount  of 

ALC  Tai 
Yes 

L 1 oring 

No 

Maintenance 

Information 

too  little 

3 

1 

acceptabl e 

14 

3 

Other 

Information 

_ 1 

too  much 

2 

1 

acceptable  | 

1 

15 

_ 1 

3 

Specification  Change  Notice 


Document  was  tailored:  14 
Tailoring  not  needed:  17 
Should  have  been  tailored:  3 


Amount  of 

ALC  Tai 
Yes 

L 1 oring 

1  No 

Maintenance  j 
Information 

too  little 

2 

1  ^ 

acceptabl e 

10 

3 

Other 

Information 

too  much 

1 

acceptabl e 

11 

4 
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Appendix  F:  Acronyms 


AFCOLR 

AFIT 

AFLC 

AFOTEC 

AFSC 

ALC 

ALD 

CDRL 

COTS 

CPIN 

CRISD 

CRLCMP 

CSCI 

CSOM 

DID 

DOD 

ECP 

FQT 

FSD 

FSM 

HWCI 

IDD 

IRS 


Air  Force  Coordinating  Office  for  Logistics 
Research 

Air  Force  Institute  of  Technology 
Air  Force  Logistics  Command 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  Systems  Command 

Air  Logistic  Center 

Acquisition  Logistics  Division 

Contract  Data  Requirements  List 

Commercial  Off  The  Shelf 

Computer  Program  Identification  Number 

Computer  Resources  Integrated  Support  Document 

Computer  Resources  Life  Cycle  Management  Plan 

Computer  Software  Configuration  Item 

Computer  System  Operator’s  Manual 

Data  Item  Description 

Department  Of  Defense 

Engineering  Change  Proposal 

Formal  Qualification  Test 

Full  Scale  Development 

Firmware  Support  Manual 

Hardware  Configuration  Item 

Interface  Design  Document 

Interface  Requirements  Specification 
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ITA 


Interface  Test  Adapter 


OFP  Operational  Flight  Program 

PMRT  Program  Management  Responsibility  Transfer 

SON  Specification  Change  Notice 

SCN  Survey  Control  Number 

SDD  Software  Design  Document 

SDF  Software  Development  File 

SDP  Software  Development  Plan 

SIRD  Software  Integration  Requirements  Document 

SPM  Software  Programmer's  Manual 

SPO  System  Program  Office 

SPS  Software  Product  Specification 

SRS  Software  Requirements  Specification 

SSDD  System/ Segment  Design  Document 

STD  Software  Test  Description 

STP  Software  Test  Plan 

STR  Software  Test  Report 

SUM  Software  User's  Manual 

TO  Technical  Order 

TRD  Test  Requirements  Document 

UUT  Unit  Under  Test 

VDD  Version  Description  Document 
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